spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Is there any plan to develop an application level fair scheduler?
Date Fri, 17 Jan 2014 21:35:20 GMT
Yup, this page describes the two levels of scheduling (within and across applications): http://spark.incubator.apache.org/docs/latest/job-scheduling.html


On Jan 17, 2014, at 12:21 PM, Reynold Xin <rxin@databricks.com> wrote:

> It does.
> 
> There are two scheduling levels here.
> 
> The first level is what the cluster manager does. The standalone cluster
> manager for Spark only supports FIFO at the moment at the level of
> applications.
> 
> Regarding Spark itself. Within a single Spark application, both FIFO and
> fair scheduling are supported, regardless of what your cluster manager is
> 
> 
> 
> On Fri, Jan 17, 2014 at 12:17 PM, Evan Chan <ev@ooyala.com> wrote:
> 
>> What is the reason that standalone mode doesn't support the fair scheduler?
>> Does that mean that Mesos coarse mode also doesn't support the fair
>> scheduler?
>> 
>> 
>> On Tue, Jan 14, 2014 at 8:10 PM, Matei Zaharia <matei.zaharia@gmail.com
>>> wrote:
>> 
>>> This is true for now, we didn’t want to replicate those systems. But it
>>> may change if we see demand for fair scheduling in our standalone cluster
>>> manager.
>>> 
>>> Matei
>>> 
>>> On Jan 14, 2014, at 6:32 PM, Xia, Junluan <junluan.xia@intel.com> wrote:
>>> 
>>>> Yes, Spark depends on Yarn or Mesos for application level scheduling.
>>>> 
>>>> -----Original Message-----
>>>> From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
>>>> Sent: Tuesday, January 14, 2014 9:43 PM
>>>> To: dev@spark.incubator.apache.org
>>>> Subject: Re: Is there any plan to develop an application level fair
>>> scheduler?
>>>> 
>>>> Hi, Junluan,
>>>> 
>>>> Thank you for the reply
>>>> 
>>>> but for the long-term plan, Spark will depend on Yarn and Mesos for
>>> application level scheduling in the coming versions?
>>>> 
>>>> Best,
>>>> 
>>>> --
>>>> Nan Zhu
>>>> 
>>>> 
>>>> On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:
>>>> 
>>>>> Are you sure that you must deploy spark in standalone mode?(it
>>> currently only support FIFO)
>>>>> 
>>>>> If you could setup Spark on Yarn or Mesos, then it has supported Fair
>>> scheduler in application level.
>>>>> 
>>>>> -----Original Message-----
>>>>> From: Nan Zhu [mailto:zhunanmcgill@gmail.com]
>>>>> Sent: Tuesday, January 14, 2014 10:13 AM
>>>>> To: dev@spark.incubator.apache.org (mailto:
>>> dev@spark.incubator.apache.org)
>>>>> Subject: Is there any plan to develop an application level fair
>>> scheduler?
>>>>> 
>>>>> Hi, All
>>>>> 
>>>>> Is there any plan to develop an application level fair scheduler?
>>>>> 
>>>>> I think it will have more value than a fair scheduler within the
>>> application (actually I didn’t understand why we want to fairly share the
>>> resource among jobs within the application, in usual, users submit
>>> different applications, not jobs)…
>>>>> 
>>>>> Best,
>>>>> 
>>>>> --
>>>>> Nan Zhu
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> --
>> --
>> Evan Chan
>> Staff Engineer
>> ev@ooyala.com  |
>> 
>> <http://www.ooyala.com/>
>> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><
>> http://www.twitter.com/ooyala>
>> 


Mime
View raw message