hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-7446) Implement CRC32C native code using SSE4.2 instructions
Date Fri, 12 Aug 2011 23:25:27 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated HADOOP-7446:

    Attachment: hadoop-7446.txt

New patch to fix up compilation issues on 32-bit:

The PIC workaround is only necessary on 32-bit PIC. If you try to use it on 64-bit PIC, you
end up with a subtle bug: gcc may actually allocate the "ebx" variable in the %ebx register,
and then you end up getting the wrong result. So, the fix is to only do the %ebx save off
to the stack in 32-bit mode. See http://www.technovelty.org/code/arch/pic-cas.html for a good

Given this, we now only worry about %ebx and not %rbx, since on 64-bit platforms we don't
have to save anything aside.

I've tested this code via TestDataChecksum on both 32-bit and 64-bit JVMs, and also ran a
short standalone test of the bulk_verify_crc32 under valgrind to check for any possible memory

> Implement CRC32C native code using SSE4.2 instructions
> ------------------------------------------------------
>                 Key: HADOOP-7446
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7446
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>         Attachments: hadoop-7446.txt, hadoop-7446.txt, hadoop-7446.txt
> Once HADOOP-7445 is implemented, we can get further performance improvements by implementing
CRC32C using the hardware support available in SSE4.2. This support should be dynamically
enabled based on CPU feature flags, and of course should be ifdeffed properly so that it doesn't
break the build on architectures/platforms where it's not available.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message