spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Feng Liu <liuj...@cn.ibm.com>
Subject Re: Fine-Grained Scheduler on Yarn
Date Fri, 08 Aug 2014 05:56:00 GMT
Thanks for echo on this. Possible to adjust resource based on container 
numbers? e.g to allocate more container when driver need more resources 
and return some resource by delete some container when parts of container 
already have enough cores/memory 
 
Best Regards
 
Jun Feng Liu
IBM China Systems & Technology Laboratory in Beijing



Phone: 86-10-82452683 
E-mail: liujunf@cn.ibm.com


BLD 28,ZGC Software Park 
No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 
China 
 

 



Patrick Wendell <pwendell@gmail.com> 
2014/08/08 13:10

To
Jun Feng Liu/China/IBM@IBMCN, 
cc
"dev@spark.apache.org" <dev@spark.apache.org>
Subject
Re: Fine-Grained Scheduler on Yarn






Hey sorry about that - what I said was the opposite of what is true.

The current YARN mode is equivalent to "coarse grained" mesos. There is no 
fine-grained scheduling on YARN at the moment. I'm not sure YARN supports 
scheduling in units other than containers. Fine-grained scheduling 
requires scheduling at the granularity of individual cores.


On Thu, Aug 7, 2014 at 9:43 PM, Patrick Wendell <pwendell@gmail.com> 
wrote:
The current YARN is equivalent to what is called "fine grained" mode in 
Mesos. The scheduling of tasks happens totally inside of the Spark driver.


On Thu, Aug 7, 2014 at 7:50 PM, Jun Feng Liu <liujunf@cn.ibm.com> wrote:
Any one know the answer?
Best Regards 
  
Jun Feng Liu
IBM China Systems & Technology Laboratory in Beijing 



Phone: 86-10-82452683 
E-mail: liujunf@cn.ibm.com 


BLD 28,ZGC Software Park 
No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 
China 
 

  



Jun Feng Liu/China/IBM 
2014/08/07 15:37 


To
dev@spark.apache.org, 
cc

Subject
Fine-Grained Scheduler on Yarn







Hi, there 

Just aware right now Spark only support fine grained scheduler on Mesos 
with MesosSchedulerBackend. The Yarn schedule sounds like only works on 
coarse-grained model. Is there any plan to implement fine-grained 
scheduler for YARN? Or there is any technical issue block us to do that.
Best Regards 
  
Jun Feng Liu
IBM China Systems & Technology Laboratory in Beijing 



Phone: 86-10-82452683 
E-mail: liujunf@cn.ibm.com 


BLD 28,ZGC Software Park 
No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 
China 
 

  



Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message