spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "donhoff_h" <>
Subject how to register CompactBuffer in Kryo
Date Fri, 28 Aug 2015 15:25:25 GMT
Hi, all

I wrote a spark program which uses the Kryo serialization. When I count a rdd which type is
RDD[(String,String)], it reported an Exception like the following :
* Class is not registered: org.apache.spark.util.collection.CompactBuffer[]
* Note: To register this class use: kryo.register(org.apache.spark.util.collection.CompactBuffer[].class);

I found that CompactBuffer is a private class that I could not use the API sparkConf.registerKryoClasses
to register it. Although I found a solution from Andras Barjak that is using the statement
: kryo.register(ClassTag(Class.forName("org.apache.spark.util.collection.CompactBuffer")).wrap.runtimeClass),
I tried it with sparkConfg.registerKryoClasses, and only got another Exception :
* Class is not registered: scala.reflect.ManifestFactory$$anon$1

Could anybody tell me :
1.Why should I register CompactBuffer[] ? My program never used it.
2.How to solve this problem?

Many Thanks!
View raw message