spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rubén Berenguel <rbereng...@gmail.com>
Subject Re: scheduler braindump: architecture, gotchas, etc.
Date Mon, 04 Feb 2019 17:01:21 GMT
Thanks Imran, will definitely give it a look (even if just out of sheer
interest on how the sausage is done)

R


On 4 February 2019 at 17:59:33, John Zhuge (jzhuge@apache.org) wrote:

Thanks Imran!

On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid <irashid@cloudera.com.invalid>
wrote:

> The scheduler has been pretty error-prone and hard to work on, and I feel
> like there may be a dwindling core of active experts.  I'm sure its very
> discouraging to folks trying to make what seem like simple changes, and
> then find they are in a rats nest of complex issues they weren't
> expecting.  But for those who are still trying, THANK YOU!  more
> involvement and more folks becoming experts is definitely needed.
>
> I put together a doc going over the architecture of the scheduler, and
> things I've seen us get bitten by in the past.  Its sort of a brain dump,
> but I'm hopeful it'll help orient new folks to the scheduler.  I also hope
> more experts will chime in -- there are places in the doc I know I've
> missed things, and called that out, but there are probably even more that
> should be discussed, & mistakes I've made.  All input welcome.
>
>
> https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing
>


--
John Zhuge

Mime
View raw message