spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ognen Duzlevski <og...@plainvanillagames.com>
Subject Re: Do all classes involving RDD operation need to be registered?
Date Fri, 28 Mar 2014 18:48:03 GMT
There is also this quote from the Tuning guide 
(http://spark.incubator.apache.org/docs/latest/tuning.html):
" Finally, if you don't register your classes, Kryo will still work, but 
it will have to store the full class name with each object, which is 
wasteful."

It implies that you don't really have to register your classes with 
Kryo. However, what kind of waste are we talking about? :)
Ognen

On 3/28/14, 12:10 PM, Debasish Das wrote:
>
> Classes are serialized and sent to all the workers as akka msgs....
>
> singletons and case classes I am not sure if they are javaserialized 
> or kryoserialized by default....
>
> But definitely your own classes if serialized by kryo will be much 
> efficient.....there is an comparison that Matei did for all the 
> serialization options and kryo was fastest at that time....
>
> Hi,
>
> I am sorry if this has been asked before. I found that if I wrapped up 
> some
> methods in a class with parameters, spark will throw "Task 
> Nonserializable"
> exception; however if wrapped up in an object or case class without
> parameters, it will work fine. Is it true that all classes involving RDD
> operation should be registered so that SparkContext could recognize them?
>
> Thanks a lot!
>
>
>
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/Do-all-classes-involving-RDD-operation-need-to-be-registered-tp3439.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.


Mime
View raw message