hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-7783) Add validation step to ensure constraints are not violated due to order in which a request is processed
Date Sun, 21 Jan 2018 10:11:00 GMT
Arun Suresh created YARN-7783:

             Summary: Add validation step to ensure constraints are not violated due to order
in which a request is processed
                 Key: YARN-7783
                 URL: https://issues.apache.org/jira/browse/YARN-7783
             Project: Hadoop YARN
          Issue Type: Sub-task
            Reporter: Arun Suresh
            Assignee: Arun Suresh

When the algorithm has placed a container on a node, allocation tags are added to the node
if the constraint is satisfied, But depending on the order in which the algorithm sees the
request, it is possible that a constraint that happen to be valid during placement of an earlier-seen
request, might not be valid after all subsequent requests have been placed.

For eg:
Assume nodes n1, n2, n3, n4 and n5
Consider the 2 constraints:
# *foo* -> anti-affinity with *foo*
# *bar* -> anti-affinity with *foo*

And 2 requests
# req1: NumAllocations = 4, allocTags = [foo]
# req2: NumAllocations = 1, allocTags = [bar]

If *req1* is seen first, the algorithm can place the 4 containers in n1, n2, n3 and n4. And
when it gets to *req2*, it will see that 4 nodes have the *foo* tag and will place it on n5.
But if *req2* is seen first, then *bar* tag will be placed on any node, since no node will
at that point have *foo*, and then when it gets to *req1*, since *foo* has no anti-affinity
with *bar*, the algorithm can end up placing *foo* on a node with *bar* violating the second

To prevent the above, we need a validation step: after the placements for a batch of requests
are made, then for each req, we remove its tags from the node and try to see of constraints
are still satisfied if the tag were to be added back on the node.

When applied to the example above, after the algorithm has run through *req2* and then *req1*,
we remove the *bar* tag from the node and try to add it back on the node. This time, constraint
satisfaction will fail, since there is now a *foo* tag on the node and *bar* cannot be added.
The algorithm will then retry placing *req2* on another node.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message