spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Error related to serialisation in spark streaming
Date Wed, 04 Jun 2014 11:44:43 GMT
static inner classes do not refer to the outer class. Often people
declare them non-static by default when it's unnecessary -- a
Comparator class is typically a great example. Anonymous inner classes
declared inside a method are another example, but there again they can
be refactored into named static inner classes.

I don't think the closure cleaner should remove this reference, unless
it somehow knows it isn't used. It could be used, that is. If it's not
used, it should just not exist in the byte code to begin with.

On Wed, Jun 4, 2014 at 12:29 PM, Mayur Rustagi <> wrote:
> I had issues around embedded functions here's what I have figured. Every
> inner class actually contains a field referencing the outer class. The
> anonymous class actually has a this$0 field referencing the outer class, and
> thus why
> Spark is trying to serialize Outer class.
> In the Scala API, the closure (which is really just implemented as anonymous
> classes) has a field called "$outer", and Spark uses a "closure cleaner"
> that goes into the anonymous class to remove the $outer field if it is not
> used in the closure itself. In Java, the compiler generates a field called
> "this$0", and thus the closure cleaner doesn't find it and can't "clean" it
> properly.
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> @mayur_rustagi
> On Wed, Jun 4, 2014 at 4:18 PM, nilmish <> wrote:
>> The error is resolved. I was using a comparator which was not serialised
>> because of which it was throwing the error.
>> I have now switched to kryo serializer as it is faster than java
>> serialser.
>> I have set the required config
>> conf.set("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer");
>> conf.set("spark.kryo.registrator", "MyRegistrator");
>> and also in MyRegistrator class I have registered all the classes I am
>> serialising.
>> How can I confirm that my code is actually using kryo serialiser and not
>> java serialiser now ?
>> PS : It seems like my code is still not using kryo serialiser.
>> --
>> View this message in context:
>> Sent from the Apache Spark User List mailing list archive at

View raw message