hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chongyuanhuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-6766) Spill can fail with bad call to Random
Date Mon, 01 Sep 2014 06:14:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117066#comment-14117066

chongyuanhuang commented on HADOOP-6766:

May be, It's taken by something about "local dirs". But how to solve this problem?

> Spill can fail with bad call to Random
> --------------------------------------
>                 Key: HADOOP-6766
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6766
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Peter Arthur Ciccolo
>            Priority: Minor
> java.lang.IllegalArgumentException: n must be positive
>         at java.util.Random.nextInt(Random.java:250)
>         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:243)
>         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:289)
>         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>         at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1129)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.Child.main(Child.java:159)
> confChanged assumes that the list of dirs it creates (LocalDirAllocator.java:215) has
at least one element in it by the end of the function. If, for each local dir, either the
conditional on line 221 is false, or the call to DiskChecker.checkDir() throws an exception,
this assumption will not hold. In this case, dirIndexRandomizer.nextInt() is called on the
number of elements in dirs, which is 0. Since dirIndexRandomizer (195) is an instance of Random(),
it needs a positive (non-zero) argument to nextInt().

This message was sent by Atlassian JIRA

View raw message