hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuqi Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-7872) labeled node cannot be used to satisfy locality specified request
Date Thu, 01 Feb 2018 13:17:00 GMT
Yuqi Wang created YARN-7872:

             Summary: labeled node cannot be used to satisfy locality specified request
                 Key: YARN-7872
                 URL: https://issues.apache.org/jira/browse/YARN-7872
             Project: Hadoop YARN
          Issue Type: Bug
          Components: capacity scheduler, capacityscheduler, resourcemanager
    Affects Versions: 2.7.2
            Reporter: Yuqi Wang
            Assignee: Yuqi Wang
             Fix For: 2.7.2, 2.8.0

labeled node (i.e. node with 'not empty' node label) cannot be used to satisfy locality specified
request (i.e. container request with 'not ANY' resource name and the relax locality is false).

For example:

The node with available resource:

[Resource: [MemoryMB: [100] CpuNumber: [12]] {color:#14892c}NodeLabel: [persistent]{color}
{color:#f79232}HostName: \{SRG}{color} RackName: \{/default-rack}]

The container request:
[Priority: [1] Resource: [MemoryMB: [1] CpuNumber: [1]] {color:#14892c}NodeLabel: [null]{color}
{color:#f79232}HostNames: \{SRG}{color} RackNames: {} {color:#59afe1}RelaxLocality: [false]{color}]


Current RM capacity scheduler's behaiour is:
The node cannot allocate container for the request because of the node label not matched
in the leaf queue assign container.

However, node locality and node label should be two orthogonal dimensions to select candidate
nodes for container request. And the node label matching should only be executed for container
request with ANY resource name, since only this kind of container request is allowed to have
'not empty' node label.

So, for container request with 'not ANY' resource name (besides, it should not have node
label), we should use resource name to match with the node instead of node label to match
with the node. And it should be safe, since the node which is not accessible for the queue
will not be sent in the leaf queue.

Attachment is the fix according to this principle, please help to review.

Without it, we cannot use locality to request container within these labeled nodes.

If the fix is acceptable, we should also recheck whether the same issue happens in trunk.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message