spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Du Li <l...@yahoo-inc.com.INVALID>
Subject Re: How to use more executors
Date Thu, 12 Mar 2015 00:50:21 GMT
Is it possible to extend this PR further (or create another PR) to allow for per-node configuration
of workers? 
There are many discussions about heterogeneous spark cluster. Currently configuration on master
will override those on the workers. Many spark users have the need for having machines with
different cpu/memory capacities in the same cluster.
Du 

     On Wednesday, January 21, 2015 3:59 PM, Nan Zhu <zhunanmcgill@gmail.com> wrote:
   

 …not sure when will it be reviewed…
but for now you can work around by allowing multiple worker instances on a single machine 
http://spark.apache.org/docs/latest/spark-standalone.html
search SPARK_WORKER_INSTANCES
Best, 
-- Nan Zhuhttp://codingcat.me On Wednesday, January 21, 2015 at 6:50 PM, Larry Liu wrote:

 Will  SPARK-1706 be included in next release?
On Wed, Jan 21, 2015 at 2:50 PM, Ted Yu <yuzhihong@gmail.com> wrote:

Please see SPARK-1706
On Wed, Jan 21, 2015 at 2:43 PM, Larry Liu <larryliu05@gmail.com> wrote:

I tried to submit a job with  --conf "spark.cores.max=6"  or --total-executor-cores 6
on a standalone cluster. But I don't see more than 1 executor on each worker. I am wondering
how to use multiple executors when submitting jobs.
Thankslarry



 
  
 

   
Mime
View raw message