spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Sripati <>
Subject Number of executors and tasks
Date Wed, 26 Nov 2014 12:24:02 GMT

I am running Spark in the stand alone mode.

1) I have a file of 286MB in HDFS (block size is 64MB) and so is split into
5 blocks. When I have the file in HDFS, 5 tasks are generated and so 5
files in the output. My understanding is that there will be a separate
partition for each block and there will be a separate task for each
partition. This makes sense why I see 5 files in the output.

When I put the same file in local file system (not HDFS), I see 9 files in
the output. I am curious why it is 9?

2) With the file in HDFS and local file system, I see a single
CoarseGrainedExecutorBackend when I run the jps command. Why is it one
executor process and how do we configure the number of executor process?


View raw message