spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: Error related to serialisation in spark streaming
Date Wed, 04 Jun 2014 11:29:49 GMT
I had issues around embedded functions here's what I have figured. Every
inner class actually contains a field referencing the outer class. The
anonymous class actually has a this$0 field referencing the outer class,
and thus why
Spark is trying to serialize Outer class.

In the Scala API, the closure (which is really just implemented
as anonymous classes) has a field called "$outer", and Spark uses a
"closure cleaner" that goes into the anonymous class to remove the $outer
field if it is not used in the closure itself. In Java, the compiler
generates a field called "this$0", and thus the closure cleaner doesn't
find it and can't "clean" it properly.



Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Wed, Jun 4, 2014 at 4:18 PM, nilmish <nilmish.iit@gmail.com> wrote:

> The error is resolved. I was using a comparator which was not serialised
> because of which it was throwing the error.
>
> I have now switched to kryo serializer as it is faster than java serialser.
> I have set the required config
>
> conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
> conf.set("spark.kryo.registrator", "MyRegistrator");
>
> and also in MyRegistrator class I have registered all the classes I am
> serialising.
>
> How can I confirm that my code is actually using kryo serialiser and not
> java serialiser now ?
>
> PS : It seems like my code is still not using kryo serialiser.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Error-related-to-serialisation-in-spark-streaming-tp6801p6904.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message