spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixcheun...@hotmail.com>
Subject Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
Date Wed, 11 Sep 2019 16:37:54 GMT
+1

________________________________
From: Thomas graves <tgraves@apache.org>
Sent: Wednesday, September 4, 2019 7:24:26 AM
To: dev <dev@spark.apache.org>
Subject: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Hey everyone,

I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
resource configuration and scheduling

This is for supporting stage level resource configuration and
scheduling.  The basic idea is to allow the user to specify executor
and task resource requirements for each stage to allow the user to
control the resources required at a finer grain. One good example here
is doing some ETL to preprocess your data in one stage and then feed
that data into an ML algorithm (like tensorflow) that would run as a
separate stage.  The ETL could need totally different resource
requirements for the executors/tasks than the ML stage does.

The text for the SPIP is in the jira description:

https://issues.apache.org/jira/browse/SPARK-27495

I split the API and Design parts into a google doc that is linked to
from the jira.

This vote is open until next Fri (Sept 13th).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Thanks,
Tom

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message