hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4863) Adding aggregationWaitMap for node-level combiner.
Date Mon, 10 Dec 2012 05:17:21 GMT
Tsuyoshi OZAWA created MAPREDUCE-4863:

             Summary: Adding aggregationWaitMap for node-level combiner.
                 Key: MAPREDUCE-4863
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4863
             Project: Hadoop Map/Reduce
          Issue Type: Sub-task
          Components: applicationmaster
    Affects Versions: 3.0.0
            Reporter: Tsuyoshi OZAWA
            Assignee: Tsuyoshi OZAWA

To manage node/rack-level combining, MRAppMaster needs to have a management information about
outputs of completed MapTasks to be aggregated.  AggregationWaitMap is used so that MRAppMaster
decides whether or not MapTasks start to combine local MapOutputFiles.

AggregationWaitMap is a abstraction class of ConcurrentHashMap<String, ArrayList<TaskAttemptCompletionEvent>.
These Events are candidate files to be aggregated.

When MapTasks are completed, MRAppMaster buffer TaskAttemptCompletionEvent into AggregationWaitMap
to delay reducers' fethcing outputs from mappers until node-level aggregation are finished.
 After node-level aggreagtion, MRAppMaster write back mapAttemptCompletionEvents, to restart
reducers' feching outputs from mappers.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message