hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roger Hoover (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-412) FifoScheduler incorrectly checking for node locality
Date Thu, 21 Feb 2013 18:08:12 GMT
Roger Hoover created YARN-412:
---------------------------------

             Summary: FifoScheduler incorrectly checking for node locality
                 Key: YARN-412
                 URL: https://issues.apache.org/jira/browse/YARN-412
             Project: Hadoop YARN
          Issue Type: Bug
          Components: scheduler
            Reporter: Roger Hoover
            Priority: Minor


In the FifoScheduler, the assignNodeLocalContainers method is checking if the data is local
to a node by searching for the nodeAddress of the node in the set of outstanding requests
for the app.  This seems to be incorrect as it should be checking hostname instead.  The offending
line of code is 455:

application.getResourceRequest(priority, node.getRMNode().getNodeAddress());

Requests are formated by hostname (e.g. host1.foo.com) where as node addresses are a concatenation
of hostname and command port (e.g. host1.foo.com:1234)

In the CapacityScheduler, it's done using hostname.  See LeafQueue.assignNodeLocalContainers,
line 1129

application.getResourceRequest(priority, node.getHostName());

Note that this but does not affect the actual scheduling decisions by the FifoScheduler because
even though it incorrect determines that a request is not local to the node, it will still
schedule the request immediately because it's rack-local.  However, this bug may be adversely
affecting the reporting of job status by underreporting the number of tasks that were node
local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message