hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar Vadali (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-2352) RAID blockfixer can use a heuristic to find unfixable files
Date Wed, 02 Mar 2011 21:13:36 GMT
RAID blockfixer can use a heuristic to find unfixable files 

                 Key: MAPREDUCE-2352
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2352
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: contrib/raid
            Reporter: Ramkumar Vadali
            Assignee: Ramkumar Vadali
            Priority: Minor

It is possible to have corrupt files that were never RAIDed. In such a case, there is no use
in trying to submit a block fixer job for that file. The RAID code has the function filterUnfixableSourceFiles()
that checks for the presence of parity files for each source file. This is too expensive,
since a lot of the parity files can be HARed. Instead, we can use a heuristic where we just
check for the presence of the parent directory in the parity space. If the parent directory
is absent, the parity file cannot be present, and the source file would be unfixable. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message