spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <>
Subject Re: Error related to serialisation in spark streaming
Date Wed, 04 Jun 2014 11:29:49 GMT
I had issues around embedded functions here's what I have figured. Every
inner class actually contains a field referencing the outer class. The
anonymous class actually has a this$0 field referencing the outer class,
and thus why
Spark is trying to serialize Outer class.

In the Scala API, the closure (which is really just implemented
as anonymous classes) has a field called "$outer", and Spark uses a
"closure cleaner" that goes into the anonymous class to remove the $outer
field if it is not used in the closure itself. In Java, the compiler
generates a field called "this$0", and thus the closure cleaner doesn't
find it and can't "clean" it properly.

Mayur Rustagi
Ph: +1 (760) 203 3257
@mayur_rustagi <>

On Wed, Jun 4, 2014 at 4:18 PM, nilmish <> wrote:

> The error is resolved. I was using a comparator which was not serialised
> because of which it was throwing the error.
> I have now switched to kryo serializer as it is faster than java serialser.
> I have set the required config
> conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
> conf.set("spark.kryo.registrator", "MyRegistrator");
> and also in MyRegistrator class I have registered all the classes I am
> serialising.
> How can I confirm that my code is actually using kryo serialiser and not
> java serialiser now ?
> PS : It seems like my code is still not using kryo serialiser.
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at

View raw message