hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oliver Hummel (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-7085) while loop in writePartitionFile method does not make sense
Date Sun, 22 Apr 2018 15:21:00 GMT
Oliver Hummel created MAPREDUCE-7085:

             Summary: while loop in writePartitionFile method does not make sense
                 Key: MAPREDUCE-7085
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7085
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2
    Affects Versions: 2.8.3
            Reporter: Oliver Hummel

After getting the split points are out of order exception from TotalOrderPartitioner, I dug
into the source of the org.apache.hadoop.mapreduce.lib.partition.InputSampler class and found
that the while loop in line 335 is never entered.

The reason is that the variable last is always smaller than k and the loop condition says
that last must be larger or equal than k.

I am not completely sure of the initial purpose of this loop, if it is what I assume, namely
reducing the occurrences of identical split points, then I would change it like so:

            while (last != -1 && k > last && comparator.compare(samples[last],
samples[k]) == 0)

{                 --k;             }

However, this only slightly mitigates the problem, since a highly skewed distribution of keys
still might lead to identical split points so that potentially further measures might be necessary?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

View raw message