hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Suresh <arun.sur...@gmail.com>
Subject Re: [DISCUSS] Merge FairScheduler preemption overhaul (YARN-4752) to trunk
Date Wed, 16 Nov 2016 19:06:57 GMT
+1 from me too.

minor nit:
* In FSAppAttempt, maybe rename addPreemption(container) /
removePreemption(container) to something better.

Also, had an offline discussion with Karthik. Suggested to maybe experiment
with replacing the PreemptionThread with a synchronous preemption approach.
For eg. Maybe keep adding to the FSStarvedApps and when it reaches a
threshold, trigger a preemption. It would likely avoid race conditions and
reduce synchronization costs.. but am ok with trying this out and
experimenting in follow up JIRAs.


On Tue, Nov 15, 2016 at 1:43 PM, Daniel Templeton <daniel@cloudera.com>

> +1 from me, but no surprise since I was the branch reviewer.
> Aside from being functionally superior to the existing code, this branch
> is also cleaner and better tested code.  We did our best to make sure that
> the patches were high quality and all the testing bases were covered.
> To add to Karthik's points, YARN-5819 was committed early this morning,
> and the JIRA for his TODO bullet is YARN-5885.
> Daniel
> On 11/10/16 6:19 PM, Karthik Kambatla wrote:
>> Forgot to mention
>>     - All the patches were reviewed before getting committed to the
>> branch.
>>     - The changes are strictly internal to the scheduler. Most of it is
>>     limited to FairScheduler with minimal changes adding helper methods to
>>     common scheduler code.
>> On Thu, Nov 10, 2016 at 4:46 PM, Karthik Kambatla <kasha@cloudera.com>
>> wrote:
>> Hi folks
>>> We have been working on overhauling FairScheduler preemption on branch
>>> YARN-4572. It is close to being ready for merge to trunk:
>>>     1. Preemption considers individual ResourceRequests to satisfy.
>>>     (YARN-5605)
>>>     2. Preemption now works within a leaf queue and across sibling leaf
>>>     queues. (YARN-5605)
>>>     3. Comprehensive unit tests for app starvation and preemption -
>>>     minshare and fairshare. (YARN-5783 and YARN-5819, the latter is
>>> close to
>>>     commit)
>>>     4. TODO: Clean up the TODOs to remove my initials and replace them
>>>     with appropriate JIRAs.
>>> There are some unresolved sub-tasks in the umbrella JIRA, but none of
>>> them
>>> are regressions in the new implementation.
>>> I just uploaded the cumulative patch to YARN-4752 for Jenkins
>>> verification
>>> and will follow up on any issues that come up.
>>> Would like to hear your thoughts on the merge.
>>> Thanks
>>> Karthik
>>> PS: Post facto, I feel a feature branch was unnecessary for this work.
>>> Github PRs with multiple commits for ease of review would have been
>>> enough.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message