spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran Rashid <>
Subject planning & discussion for larger scheduler changes
Date Fri, 24 Mar 2017 15:41:01 GMT
Kay and I were discussing some of the  bigger scheduler changes getting
proposed lately, and realized there is a broader discussion to have with
the community, outside of any single jira.  I'll start by sharing my
initial thoughts, I know Kay has thoughts on this too, but it would be good
to input from everyone.

In particular, SPARK-14649 & SPARK-13669 have got me thinking.  These are
proposed changes in behavior that are not fixes for *correctness* in fault
tolerance, but to improve the performance when there faults.  The changes
make some intuitive sense, but its also hard to judge whether they are
necessarily better; its hard to verify the correctness of the changes; and
its hard to even know that we haven't broken the old behavior (because of
how brittle the scheduler seems to be).

So I'm wondering:

1) in the short-term, can we find ways to get these changes merged, but
turned off by default, in a way that we feel confident won't break existing

2) a bit longer-term -- should we be considering bigger rewrites to the
scheduler?  Particularly, to improve testability?  eg., maybe if it was
rewritten to more completely follow the actor model and eliminate shared
state, the code would be cleaner and more testable.  Or maybe this is a
crazy idea, and we'd just lose everything we'd learned so far and be stuck
fixing the as many bugs in the new version.


View raw message