hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Finnessy (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1471) FileOutputCommitter does not safely clean up it's temporary files
Date Tue, 09 Feb 2010 15:25:28 GMT
FileOutputCommitter does not safely clean up it's temporary files

                 Key: MAPREDUCE-1471
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1471
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 0.20.1
            Reporter: Jim Finnessy

When the FileOutputCommitter cleans up during it's cleanupJob method, it potentially deletes
the temporary files of other concurrent jobs.

Since all the temporary files for all concurrent jobs are written to working_path/_temporary/
any concurrent tasks that have the same working_path will remove all currently executing jobs
when it removes working_path/_temporary during job cleanup.

If the file name output is guaranteed by the client application to be unique, the temporary
files/directories should also be guaranteed to be unique to avoid this problem. Suggest modifying
cleanupJob to only remove files that it created itself.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message