hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4922) Request with multiple data local nodes can cause NPE in AppSchedulingInfo
Date Tue, 08 Jan 2013 00:06:12 GMT
Sandy Ryza created MAPREDUCE-4922:

             Summary: Request with multiple data local nodes can cause NPE in AppSchedulingInfo
                 Key: MAPREDUCE-4922
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4922
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 2.0.2-alpha
            Reporter: Sandy Ryza
            Assignee: Sandy Ryza

With the way that the schedulers work, each request for a container on a node must consist
of 3 ResourceRequests - one on the node, one on the rack, and one with *.

AppSchedulingInfo tracks the outstanding requests.  When a node is assigned a node-local container,
allocateNodeLocal decrements the outstanding requests at each level - node, rack, and *. 
If the rack requests reach 0, it removes the mapping.

A mapreduce task with multiple data local nodes submits multiple container requests, one for
each node.  It also submits one for each unique rack, and one for *.  If there are fewer unique
racks than data local nodes, this means that fewer rack-local ResourceRequests will be submitted
than node-local ResourceRequests, so the rack-local mapping will be deleted before all the
node-local requests are allocated and an NPE will come up the next time a node-local request
from that rack is allocated.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message