hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4912) Investigate ways to clean up double job commit prevention
Date Fri, 04 Jan 2013 15:30:12 GMT
Robert Joseph Evans created MAPREDUCE-4912:

             Summary: Investigate ways to clean up double job commit prevention
                 Key: MAPREDUCE-4912
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4912
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2
            Reporter: Robert Joseph Evans

Once MAPREDUCE-4819 goes in it fixes the issue where an OutputCommiter can double commit a
job.  So that the output will never be touched after the job informs externally of success
or failure.

The code and design could potentially use some cleanup and refactoring.

Issues brought up that should be investigated include:

# reporting KILL for killed jobs if they crash after the kill happens instead of error.
# using the job history log for recording the commit status instead of separate external files
in HDFS.
# Placing the recovery/retry logic in the commit handler instead of the MRAppMaster, and having
the recovery service replay the logs as it normally does for recovery.

This is not meant to be things that must be done, but alternatives that might clean up the

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message