hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-3512) Batch jobHistory disk flushes
Date Tue, 06 Dec 2011 00:33:40 GMT
Batch jobHistory disk flushes

                 Key: MAPREDUCE-3512
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mr-am, mrv2
    Affects Versions: 0.23.0
            Reporter: Siddharth Seth

The mr-am flushes each individual job history event to disk for AM recovery. The history even
handler ends up with a significant backlog for tests like MAPREDUCE-3402. 
History events could be batched up based on num records / time / TaskFinishedEvents to reduce
the number of DFS writes - with the potential drawback of having to rerun some tasks during
AM recovery.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message