hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Botong Huang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-7631) ResourceRequest with different Capacity (Resource) overrides each other in RM
Date Sat, 09 Dec 2017 00:39:00 GMT
Botong Huang created YARN-7631:

             Summary: ResourceRequest with different Capacity (Resource) overrides each other
in RM
                 Key: YARN-7631
                 URL: https://issues.apache.org/jira/browse/YARN-7631
             Project: Hadoop YARN
          Issue Type: Bug
            Reporter: Botong Huang

Today in AMRMClientImpl, the ResourceRequests (RR) are kept as: RequestId -> Priority ->
ResourceName -> ExecutionType -> Resource (Capacity) -> ResourceRequestInfo (the
actual RR). 

This means that only RRs with the same (requestId, priority, resourcename, executionType,
resource) will be grouped and aggregated together. 

While in RM side, the mapping is SchedulerRequestKey (RequestId, priority) -> LocalityAppPlacementAllocator
(ResourceName -> RR). 

The issue is that in RM side Resource is not in the key to the RR at all. (Note that executionType
is also not in the RM side, but it is fine because RM handles it separately as container update
requests.) This means that under the same value of (requestId, priority, resourcename), RRs
with different Resource values will be grouped together and override each other in RM. As
a result, some of the container requests are lost and will never be allocated. Furthermore,
since the two RRs are kept under different keys in AMRMClient side, allocation of RR1 will
only trigger cancel for RR1, the pending RR2 will not get resend as well. 

I’ve attached an unit test (resourcebug.patch) which is failing in trunk to illustrate this

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message