spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: Division of work between master, worker, executor and driver
Date Mon, 27 Jan 2014 02:38:15 GMT
Correct.


On Sun, Jan 26, 2014 at 6:30 PM, Manoj Samel <manojsameltech@gmail.com>wrote:

> Yes, that's what I meant (thanks for the correction).
>
> From the tests run, it seems best is to start workers with default mem (or
> bit higher) and give much more memory/most of the memory to executors;
> since most of the work will be done in executor jvm and the worker jvm
> seems more like node manager for that node.
>
>
> On Sat, Jan 25, 2014 at 6:32 AM, Archit Thakur <archit279thakur@gmail.com>wrote:
>
>>
>>
>>
>> On Fri, Jan 24, 2014 at 11:29 PM, Manoj Samel <manojsameltech@gmail.com>wrote:
>>
>>> On cluster with HDFS + Spark (in standalone deploy mode), there is a
>>> master node + 4 worker nodes. When a spark-shell connects to master, it
>>> creates 4 executor JVMs on each of the 4 worker nodes.
>>>
>>
>> No, It creates 1 (4 in total) executor JVM on each of the 4 worker nodes.
>>
>>>
>>> When the application reads a HDFS files and does computations in RDDs,
>>> what work gets done on master, worker, executor and driver  ?
>>>
>>> Thanks,
>>>
>>
>>
>

Mime
View raw message