hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Palaniappan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-8118) Better utilize gracefully decommissioning node managers
Date Wed, 04 Apr 2018 23:12:00 GMT
Karthik Palaniappan created YARN-8118:

             Summary: Better utilize gracefully decommissioning node managers
                 Key: YARN-8118
                 URL: https://issues.apache.org/jira/browse/YARN-8118
             Project: Hadoop YARN
          Issue Type: Sub-task
          Components: yarn
    Affects Versions: 2.8.2
         Environment: * Google Compute Engine (Dataproc)
 * Java 8
 * Hadoop 2.8.2 using client-mode graceful decommissioning
            Reporter: Karthik Palaniappan

Proposal design doc with background + details: (will link – please comment on that doc)

tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications to complete
before shutting down, but they cannot run new containers from those in-progress applications. This
is wasteful, particularly in environments where you are billed by resource usage (e.g. EC2).

Proposal: YARN should schedule containers from in-progress applications on DECOMMISSIONING
nodes, but should still avoid scheduling containers from new applications. That will make in-progress
applications complete faster and let nodes decommission faster. Overall, this should be cheaper.

I have a working patch without unit tests (will attach) that's surprisingly just a few real
lines of code. If folks are happy with the proposal, I'll write unit tests and also write
a patch targeted at trunk.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message