spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran Rashid <>
Subject Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling
Date Mon, 04 Mar 2019 20:52:24 GMT
On Sun, Mar 3, 2019 at 6:51 PM Xiangrui Meng <> wrote:

> On Sun, Mar 3, 2019 at 10:20 AM Felix Cheung <>
> wrote:
>> IMO upfront allocation is less useful. Specifically too expensive for
>> large jobs.
> This is also an API/design discussion.

I agree with Felix -- this is more than just an API question.  It has a
huge impact on the complexity of what you're proposing.  You might be
proposing big changes to a core and brittle part of spark, which is already
short of experts.

I don't see any value in having a vote on "does feature X sound cool?"  We
have to evaluate the potential benefit against the risks the feature brings
and the continued maintenance cost.  We don't need super low-level details,
but we have to a sketch of the design to be able to make that tradeoff.

View raw message