hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Jie (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6459) native task crashes when merging spilled file on ppc64
Date Fri, 21 Aug 2015 10:22:45 GMT
Tao Jie created MAPREDUCE-6459:

             Summary: native task crashes when merging spilled file on ppc64
                 Key: MAPREDUCE-6459
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6459
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 2.6.0
         Environment: Linux version 2.6.32-431.el6.ppc64
            Reporter: Tao Jie
         Attachments: ppc64_error.txt

when running native task on ppc64´╝îmerging spilled files fails since we could not deserialize
local spill file correctly.
Function readVLong in WritableUtils.h and Buffers.h, we try to compare a char with a number
and convert a char to int64_t. It does not work correctly on ppc64 since char definition is
different between ppc64 and x86 platform. On x86 platform char is defined as signed number
while on ppc64 char is unsigned. As a result, we write EOF marker [-1, -1] at the end of spill
partition, but deserialize chars as [255, 255].

This message was sent by Atlassian JIRA

View raw message