spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Seluka <psel...@qubole.com>
Subject Re: API to add/remove containers inside an application
Date Fri, 05 Sep 2014 06:18:24 GMT
Mailed our list - will send it to Spark Dev


On Fri, Sep 5, 2014 at 11:28 AM, Rajat Gupta <rgupta@qubole.com> wrote:

> +1 on this. First step to more automated autoscaling of spark application
> master...
>
>
> On Fri, Sep 5, 2014 at 12:56 AM, Praveen Seluka <pseluka@qubole.com>
> wrote:
>
>> +user
>>
>>
>>
>> On Thu, Sep 4, 2014 at 10:53 PM, Praveen Seluka <pseluka@qubole.com>
>> wrote:
>>
>>> Spark on Yarn has static allocation of resources.
>>> https://issues.apache.org/jira/browse/SPARK-3174 - This JIRA by Sandy
>>> is about adding and removing executors dynamically based on load. Even
>>> before doing this, can we expose and interface to add/remove executors.
>>> This will be very useful in Spark Shell.
>>>
>>> User can launch Spark Shell with just few executors. And later user
>>> wants more executors to be added, user could call some API like
>>> sc.addExecutor(count). We could similarly have removeExecutor API too. I
>>> have been thinking to work on this. Later, SPARK-3174 JIRA can add
>>> intelligence on when to add/remove depending on load. Right now, user will
>>> get control over adding and removing executor at runtime.
>>>
>>> Thoughts?
>>>
>>> Praveen
>>>
>>
>>
>

Mime
View raw message