spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rubén Berenguel <>
Subject Re: scheduler braindump: architecture, gotchas, etc.
Date Mon, 04 Feb 2019 17:01:21 GMT
Thanks Imran, will definitely give it a look (even if just out of sheer
interest on how the sausage is done)


On 4 February 2019 at 17:59:33, John Zhuge ( wrote:

Thanks Imran!

On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid <>

> The scheduler has been pretty error-prone and hard to work on, and I feel
> like there may be a dwindling core of active experts.  I'm sure its very
> discouraging to folks trying to make what seem like simple changes, and
> then find they are in a rats nest of complex issues they weren't
> expecting.  But for those who are still trying, THANK YOU!  more
> involvement and more folks becoming experts is definitely needed.
> I put together a doc going over the architecture of the scheduler, and
> things I've seen us get bitten by in the past.  Its sort of a brain dump,
> but I'm hopeful it'll help orient new folks to the scheduler.  I also hope
> more experts will chime in -- there are places in the doc I know I've
> missed things, and called that out, but there are probably even more that
> should be discussed, & mistakes I've made.  All input welcome.

John Zhuge

View raw message