spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: serialization stakeoverflow error during reduce on nested objects
Date Sat, 14 Mar 2015 05:22:58 GMT
Have you registered your class with kryo ?

See core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
and core/src/test/scala/org/apache/spark/SparkConfSuite.scala

On Fri, Mar 13, 2015 at 10:52 AM, ilaxes <ilaxes@hotmail.com> wrote:

> Hi,
>
> I'm working on a RDD of a tuple of objects which represent trees (Node
> containing a hashmap of nodes). I'm trying to aggregate these trees over
> the
> RDD.
>
> Let's take an example, 2 graphs :
> C - D - B - A - D - B - E
> F - E - B - A - D - B - F
>
> I'm spliting each graphs according to the vertex A resulting in :
> (B(1, D(1, C(1,))) , D(1, B(1, E(1,)))
> (B(1, E(1, F(1,))) , D(1, B(1, F(1,)))
>
> And I want to aggregate both graph getting :
> (B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,)))
>
> Some graph are potentially large (+4000 vertex) but I'm not supposed to
> have
> any cyclic references.
> When I run my program I get this error :
>
> java.lang.StackOverflowError
>         at
>
> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127)
>         at
>
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
>         at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
>         at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
>         at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
>         at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>         at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
>         at
>
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
>         at
>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
>         at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
>         at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
>         at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>         at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
>
> I've tried to increase the size of the stake and use the standard java
> serializer but no effect.
>
> Any hint of the reason of this error and ways to change my code to solve it
> ?
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message