spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "donhoff_h" <165612...@qq.com>
Subject how to register CompactBuffer in Kryo
Date Fri, 28 Aug 2015 15:25:25 GMT
Hi, all


I wrote a spark program which uses the Kryo serialization. When I count a rdd which type is
RDD[(String,String)], it reported an Exception like the following :
* Class is not registered: org.apache.spark.util.collection.CompactBuffer[]
* Note: To register this class use: kryo.register(org.apache.spark.util.collection.CompactBuffer[].class);



I found that CompactBuffer is a private class that I could not use the API sparkConf.registerKryoClasses
to register it. Although I found a solution from Andras Barjak that is using the statement
: kryo.register(ClassTag(Class.forName("org.apache.spark.util.collection.CompactBuffer")).wrap.runtimeClass),
I tried it with sparkConfg.registerKryoClasses, and only got another Exception :
* Class is not registered: scala.reflect.ManifestFactory$$anon$1


Could anybody tell me :
1.Why should I register CompactBuffer[] ? My program never used it.
2.How to solve this problem?


Many Thanks!
Mime
View raw message