spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <>
Subject Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
Date Wed, 11 Sep 2019 13:35:30 GMT
I am +1.

I like the proposal and it feels like it will provide what users need for
the use cases in the SPIP. Having done similar things on other projects in
the past I know this is just the beginning to get something that is truly
user and admin friendly, but this is a great start.



On Wed, Sep 4, 2019 at 9:24 AM Thomas graves <> wrote:

> Hey everyone,
> I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
> resource configuration and scheduling
> This is for supporting stage level resource configuration and
> scheduling.  The basic idea is to allow the user to specify executor
> and task resource requirements for each stage to allow the user to
> control the resources required at a finer grain. One good example here
> is doing some ETL to preprocess your data in one stage and then feed
> that data into an ML algorithm (like tensorflow) that would run as a
> separate stage.  The ETL could need totally different resource
> requirements for the executors/tasks than the ML stage does.
> The text for the SPIP is in the jira description:
> I split the API and Design parts into a google doc that is linked to
> from the jira.
> This vote is open until next Fri (Sept 13th).
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don't think this is a good idea because ...
> I'll start with my +1
> Thanks,
> Tom
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message