spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barona, Ricardo" <ricardo.bar...@intel.com>
Subject Re: Number of executors and dynamic allocation
Date Mon, 03 Apr 2017 15:12:00 GMT
Completely not a silly question, it’s actually a really good observation. I think we didn’t
update correctly the ml_ops.sh script when we added these parameters. 

What we could start discussing is whether we want dynamic allocation or if we want fixed number
of executors. I’m going to let the mic open to see what people think about this.

Thanks Giacomo.

On 4/3/17, 3:29 AM, "Giacomo Bernardi" <mino@minux.it> wrote:

    Hi,
    hope this is not a silly question. In ml_ops.sh there are:
      --num-executors ${SPK_EXEC} \
    and:
      --conf spark.dynamicAllocation.enabled=true \
    
    which trigger the warning:
      WARN spark.SparkContext: Dynamic Allocation and num executors both
    set, thus dynamic allocation disabled.
    
    Shouldn't we remove the "--num-executors" and add instead:
      --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
    ?
    
    Thanks.
    Giacomo
    

Mime
View raw message