spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject quick question
Date Thu, 01 Dec 2016 20:25:07 GMT
Assume I am running a Spark Client Program in client mode and Spark Cluster
in Stand alone mode.

I want some clarification of the following things

1. Build a DAG
2. DAG Scheduler
3. TASK Scheduler

I want to which of the above part is done by SPARK CLIENT and which of the
above parts are done by SPARK MASTER in the stand alone case?

Building a DAG clearly looks like Spark Client Program
DAG Scheduler is also in the Spark Client Program
Task Scheduler is done by the SPARK MASTER.

is this correct? Also, Does Spark Client every instruct Spark Workers
directly on what transformations to run or the communication is just
unidirectional in the sense that Spark Workers communicate to Spark client
only when returning the results ?

thanks!

Mime
View raw message