spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Fine-Grained Scheduler on Yarn
Date Fri, 08 Aug 2014 05:10:55 GMT
Hey sorry about that - what I said was the opposite of what is true.

The current YARN mode is equivalent to "coarse grained" mesos. There is no
fine-grained scheduling on YARN at the moment. I'm not sure YARN supports
scheduling in units other than containers. Fine-grained scheduling requires
scheduling at the granularity of individual cores.


On Thu, Aug 7, 2014 at 9:43 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> The current YARN is equivalent to what is called "fine grained" mode in
> Mesos. The scheduling of tasks happens totally inside of the Spark driver.
>
>
> On Thu, Aug 7, 2014 at 7:50 PM, Jun Feng Liu <liujunf@cn.ibm.com> wrote:
>
>> Any one know the answer?
>>
>> Best Regards
>>
>>
>> *Jun Feng Liu*
>> IBM China Systems & Technology Laboratory in Beijing
>>
>>   ------------------------------
>>  [image: 2D barcode - encoded with contact information] *Phone: *86-10-82452683
>>
>> * E-mail:* *liujunf@cn.ibm.com* <liujunf@cn.ibm.com>
>> [image: IBM]
>>
>> BLD 28,ZGC Software Park
>> No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193
>> China
>>
>>
>>
>>
>>
>>  *Jun Feng Liu/China/IBM*
>>
>> 2014/08/07 15:37
>>   To
>> dev@spark.apache.org,
>> cc
>>   Subject
>> Fine-Grained Scheduler on Yarn
>>
>>
>>
>> Hi, there
>>
>> Just aware right now Spark only support fine grained scheduler on Mesos
>> with MesosSchedulerBackend. The Yarn schedule sounds like only works on
>> coarse-grained model. Is there any plan to implement fine-grained scheduler
>> for YARN? Or there is any technical issue block us to do that.
>>
>> Best Regards
>>
>>
>> *Jun Feng Liu*
>> IBM China Systems & Technology Laboratory in Beijing
>>
>>   ------------------------------
>>  [image: 2D barcode - encoded with contact information] *Phone: *86-10-82452683
>>
>> * E-mail:* *liujunf@cn.ibm.com* <liujunf@cn.ibm.com>
>> [image: IBM]
>>
>> BLD 28,ZGC Software Park
>> No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193
>> China
>>
>>
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message