spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giacomo Bernardi <m...@minux.it>
Subject Number of executors and dynamic allocation
Date Mon, 03 Apr 2017 08:29:29 GMT
Hi,
hope this is not a silly question. In ml_ops.sh there are:
  --num-executors ${SPK_EXEC} \
and:
  --conf spark.dynamicAllocation.enabled=true \

which trigger the warning:
  WARN spark.SparkContext: Dynamic Allocation and num executors both
set, thus dynamic allocation disabled.

Shouldn't we remove the "--num-executors" and add instead:
  --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
?

Thanks.
Giacomo

Mime
View raw message