spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: How to set executor num on spark on yarn
Date Tue, 16 Sep 2014 10:57:21 GMT
How many cores do your machines have? --executor-cores should be the
number of cores each executor uses. Fewer cores means more executors
in general. From your data, it sounds like, for example, there are 7
nodes with 4+ cores available to YARN, and 2 more nodes with 2-3 cores
available. Hence when you ask for fewer cores per executor, more can
fit.

You may need to increase the number of cores you are letting YARN
manage if this doesn't match your expectation. For example, I'm
guessing your machines have more than 4 cores in reality.

On Tue, Sep 16, 2014 at 6:08 AM, hequn cheng <chenghequn@gmail.com> wrote:
> hi~I want to set the executor number to 16, but it is very strange that
> executor cores may affect executor num on spark on yarn, i don't know why
> and how to set executor number.
> =============================================
> ./bin/spark-submit --class com.hequn.spark.SparkJoins \
>     --master yarn-cluster \
>     --num-executors 16 \
>     --driver-memory 2g \
>     --executor-memory 10g \
>     --executor-cores 4 \
>     /home/sparkjoins-1.0-SNAPSHOT.jar
>
> The UI shows there are 7 executors
> =============================================
> ./bin/spark-submit --class com.hequn.spark.SparkJoins \
>     --master yarn-cluster \
>     --num-executors 16 \
>     --driver-memory 2g \
>     --executor-memory 10g \
>     --executor-cores 2 \
>     /home/sparkjoins-1.0-SNAPSHOT.jar
>
> The UI shows there are 9 executors
> =============================================
> ./bin/spark-submit --class com.hequn.spark.SparkJoins \
>     --master yarn-cluster \
>     --num-executors 16 \
>     --driver-memory 2g \
>     --executor-memory 10g \
>     --executor-cores 1 \
>     /home/sparkjoins-1.0-SNAPSHOT.jar
>
> The UI shows there are 9 executors
> ==============================================
> The cluster contains 16 nodes. Each node 64G RAM.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message