Hi Andrea,

What version of Spark are you using?  There were some improvements in how Spark uses Kryo in 0.9.1 and to-be 1.0 that I would expect to improve this.

Also, can you share your registrator's code?

Another possibility is that Kryo can have some difficulty serializing very large objects.  Do you have a sense of how large the serialized items in your RDD are?


On Sat, May 10, 2014 at 6:32 AM, Andrea Esposito <and1989@gmail.com> wrote:
UP, doesn't anyone know something about it? ^^

2014-05-06 12:05 GMT+02:00 Andrea Esposito <and1989@gmail.com>:

Hi there,

sorry if i'm posting a lot lately.

i'm trying to add the KryoSerializer but i receive this exception:
2014 - 05 - 06 11: 45: 23 WARN TaskSetManager: 62 - Loss was due to java.io.EOFException
at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala: 105)
at org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala: 165)
at org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala: 56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43)
at java.lang.reflect.Method.invoke(Method.java: 606)

I set the serializer as:
System.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
System.setProperty("spark.kryo.registrator", "test.TestKryoRegistrator")

With or without register my custom registrator it throws the exception.

Seems something related to broadcast.. but isn't Kryo already ok out of the box just setting it as default serializer?