hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subir S (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-3924) Partioning Failing in certain scenarios for Hadoop Streaming using Ruby
Date Sat, 25 Feb 2012 16:01:49 GMT
Partioning Failing in certain scenarios for Hadoop Streaming using Ruby

                 Key: MAPREDUCE-3924
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3924
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 0.20.2
         Environment: Linux OS
44 Node cluster
Hadoop version 0.20.3-cdhu3 is the version
            Reporter: Subir S

We noticed a wierd scenario. The partitioning option fails in scenario below. The map reduce
just aggregates values like a word count. However the key is having 2 parts.

Input Records: (<key1|key2><tab><qty1|amt1|qty2|amt2|qty3|amt3>
1201|420 1|24.0|2|26.0|0|0.0
1200|420 1|24.0|2|26.0|0|0.0
1200|420 1|25.0|3|52.0|1|55.0
1403|400 1|25.0|3|52.0|1|55.0
1201|420 1|24.0|2|26.0|0|0.0
1201|420 1|24.0|2|26.0|0|0.0
1403|400 1|25.0|3|52.0|1|55.0

Partioning option: -k1,1 and Comparator option -k1,1n -k2,2n works fine and the output we
get is expected output
1200|420 2|49.0|5|78.0|1|55.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0

However if we use
Partioning option: -k1,1 -k2,2 and Comparator option -k1,1n -k2,2n then the output is faulty.
Seems the same key does not go to same reducer in output we see duplicate records, however
this does not happen to all records. A hypothetical faulty output based on above input is:
1200|420 1|24.0|2|26.0|0|0.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0
1200|420 1|25.0|3|52.0|1|55.0

See that records with the key (1200|420) is not aggregated. This can happen only if the record
does not go to the same reducer after partioning.

Any clue why this is happening? I could not understanding what is going wrong.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message