hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5962) Support CRC32C in IFile
Date Mon, 07 Jul 2014 16:58:33 GMT
Todd Lipcon created MAPREDUCE-5962:

             Summary: Support CRC32C in IFile
                 Key: MAPREDUCE-5962
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5962
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: performance, task
    Affects Versions: 2.5.0
            Reporter: Todd Lipcon

Currently, the IFile format used by the MR shuffle checksums all data using the zlib CRC32
polynomial. If we allow use of CRC32C instead, we can get a large reduction in CPU usage by
leveraging the native hardware CRC32C implementation (approx half a second of CPU time savings
per GB checksummed).

This message was sent by Atlassian JIRA

View raw message