hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Shraibman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-2564) NullPointerException in WritableComparator
Date Fri, 03 Jun 2011 15:37:50 GMT
NullPointerException in WritableComparator

                 Key: MAPREDUCE-2564
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: task
    Affects Versions:
         Environment: java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)

            Reporter: Joseph Shraibman
            Priority: Blocker

        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

It is easy to see why this is happening.  The WritableComparator is created in JobConf line
   return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
which calls
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
    this(keyClass, false);

  protected WritableComparator(Class<? extends WritableComparable> keyClass,
      boolean createInstances) {
    this.keyClass = keyClass;
    if (createInstances) {
      key1 = newKey();
      key2 = newKey();
      buffer = new DataInputBuffer();
    } else {
      key1 = key2 = null;
      buffer = null;

key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because
buffer is null

 public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
    try {
      buffer.reset(b1, s1, l1);                   // parse key1
      buffer.reset(b2, s2, l2);                   // parse key2
    } catch (IOException e) {
      throw new RuntimeException(e);
    return compare(key1, key2);                   // compare them

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message