spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barona, Ricardo" <ricardo.bar...@intel.com>
Subject Re: Number of executors and dynamic allocation
Date Mon, 03 Apr 2017 15:19:49 GMT
I created this bug in JIRA: https://issues.apache.org/jira/browse/SPOT-136
We can continue discussing the details in there.

Thanks. 

On 4/3/17, 10:12 AM, "Barona, Ricardo" <ricardo.barona@intel.com> wrote:

    Completely not a silly question, it’s actually a really good observation. I think we
didn’t update correctly the ml_ops.sh script when we added these parameters. 
    
    What we could start discussing is whether we want dynamic allocation or if we want fixed
number of executors. I’m going to let the mic open to see what people think about this.
    
    Thanks Giacomo.
    
    On 4/3/17, 3:29 AM, "Giacomo Bernardi" <mino@minux.it> wrote:
    
        Hi,
        hope this is not a silly question. In ml_ops.sh there are:
          --num-executors ${SPK_EXEC} \
        and:
          --conf spark.dynamicAllocation.enabled=true \
        
        which trigger the warning:
          WARN spark.SparkContext: Dynamic Allocation and num executors both
        set, thus dynamic allocation disabled.
        
        Shouldn't we remove the "--num-executors" and add instead:
          --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
        ?
        
        Thanks.
        Giacomo
        
    
    

Mime
View raw message