spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Nguyen <...@adatao.com>
Subject Re: Not able to understand Exception.
Date Wed, 01 Jan 2014 22:56:35 GMT
Archit, this occurs in the ResultTask phase, triggered by the call to
sortByKey. Prior to this, your RDD would have been serialized for, e.g.,
shuffling around.

So it looks like Kryo wasn't able to deserialize some part of the RDD for
some reason, possible due to formatting incompatibility. Did you say you
wrote your own serializers?

--
Christopher T. Nguyen
Co-founder & CEO, Adatao <http://adatao.com>
linkedin.com/in/ctnguyen



On Wed, Jan 1, 2014 at 8:22 AM, Archit Thakur <archit279thakur@gmail.com>wrote:

> I have recently moved to Kryo for serialization to get better performance.
> Have written some of the serializers for my custom DS.
> What could below exception be about: (I dont see any of my code line in
> the stack trace)
>
> java.lang.ArrayIndexOutOfBoundsException: -2
>         at java.util.ArrayList.get(Unknown Source)
>         at
> com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
>         at
> com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:773)
>         at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:727)
>         at
> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:106)
>         at
> org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:101)
>         at
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
>         at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440)
>         at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:26)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>         at
> org.apache.spark.util.CompletionIterator.foreach(CompletionIterator.scala:23)
>         at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>         at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
>         at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:250)
>         at
> org.apache.spark.util.CompletionIterator.toBuffer(CompletionIterator.scala:23)
>         at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:237)
>         at
> org.apache.spark.util.CompletionIterator.toArray(CompletionIterator.scala:23)
>         at
> org.apache.spark.rdd.OrderedRDDFunctions$$anonfun$sortByKey$1.apply(OrderedRDDFunctions.scala:44)
>         at
> org.apache.spark.rdd.OrderedRDDFunctions$$anonfun$sortByKey$1.apply(OrderedRDDFunctions.scala:43)
>         at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:36)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
>         at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:29)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
>         at org.apache.spark.scheduler.ResultTask.run(ResultTask.scala:99)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>         at java.lang.Thread.run(Unknown Source)
>
> Any ideas? or Suggestions would help.
>
> Thanks,
> Archit.
>

Mime
View raw message