hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Bacsko (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-7056) Ensure that mapreduce.job.reduces is not negative
Date Tue, 20 Feb 2018 15:47:00 GMT
Peter Bacsko created MAPREDUCE-7056:

             Summary: Ensure that mapreduce.job.reduces is not negative
                 Key: MAPREDUCE-7056
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7056
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: client
            Reporter: Peter Bacsko
            Assignee: Peter Bacsko

Recently we've seen a strange problem that was related to {{mapreduce.job.reduces}} being
set to -1. If this value is negative, two things can happen:

1. If we use the old API, then the mappers will pass, but the number of reducers will be recorded
as "-1" if we open it later from JHS. This can confuse Hadoop users.

2. If use the new API, then we'll see a not-so-obvious stack trace:

2018-02-20 06:37:35,493 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map
2018-02-20 06:37:35,507 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map
2018-02-20 06:37:35,507 INFO [main] org.apache.hadoop.mapred.MapTask: kvbuffer is null. Skipping
2018-02-20 06:37:35,508 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.lang.IllegalArgumentException
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:51)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1891)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1527)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:735)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:805)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

and the job fails.

We should either fail if this property is negative or set to "0" to avoid this.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

View raw message