spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Cutler <cutl...@gmail.com>
Subject Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
Date Wed, 11 Sep 2019 18:10:20 GMT
+1 (non-binding), looks good!

On Wed, Sep 11, 2019 at 10:05 AM Ryan Blue <rblue@netflix.com.invalid>
wrote:

> +1
>
> This is going to be really useful. Thanks for working on it!
>
> On Wed, Sep 11, 2019 at 9:38 AM Felix Cheung <felixcheung_m@hotmail.com>
> wrote:
>
>> +1
>>
>> ------------------------------
>> *From:* Thomas graves <tgraves@apache.org>
>> *Sent:* Wednesday, September 4, 2019 7:24:26 AM
>> *To:* dev <dev@spark.apache.org>
>> *Subject:* [VOTE] [SPARK-27495] SPIP: Support Stage level resource
>> configuration and scheduling
>>
>> Hey everyone,
>>
>> I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
>> resource configuration and scheduling
>>
>> This is for supporting stage level resource configuration and
>> scheduling.  The basic idea is to allow the user to specify executor
>> and task resource requirements for each stage to allow the user to
>> control the resources required at a finer grain. One good example here
>> is doing some ETL to preprocess your data in one stage and then feed
>> that data into an ML algorithm (like tensorflow) that would run as a
>> separate stage.  The ETL could need totally different resource
>> requirements for the executors/tasks than the ML stage does.
>>
>> The text for the SPIP is in the jira description:
>>
>> https://issues.apache.org/jira/browse/SPARK-27495
>>
>> I split the API and Design parts into a google doc that is linked to
>> from the jira.
>>
>> This vote is open until next Fri (Sept 13th).
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don't think this is a good idea because ...
>>
>> I'll start with my +1
>>
>> Thanks,
>> Tom
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Mime
View raw message