spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hegner, Travis" <>
Subject SPARK-18689: A proposal for priority based app scheduling utilizing linux cgroups.
Date Fri, 02 Dec 2016 20:27:31 GMT

I've just created a JIRA to open up discussion of a new feature that I'd like to propose.

I'd love to get some feedback on the idea. I know that normally anything related to scheduling
or queuing automatically throws up the "hard to implement" red flags, but the proposal contains
a rather simple way to implement the concept, which delegates the scheduling logic to the
actual kernel of each worker, rather than in any spark core code. I believe this to be more
flexible and simpler to set up and maintain than dynamic allocation, and avoids the need for
any preemption type of logic.

The proposal does not contain any code. I am not (yet) familiar enough with the core spark
code to confidently create an implementation.

I appreciate your time and am looking forward to your feedback!



View raw message