spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "donhoff_h" <>
Subject Serialization Problem in Spark Program
Date Wed, 25 Mar 2015 07:44:29 GMT
Hi, experts

I wrote a very simple spark program to test the KryoSerialization function. The codes are
as following:

object TestKryoSerialization {
  def main(args: Array[String]) {
    val conf = new SparkConf()
    conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    conf.set("spark.kryo.registrationRequired","true")  //I use this statement to force checking

    val sc = new SparkContext(conf)
    val rdd = sc.textFile("hdfs://dhao.hrb:8020/user/spark/tstfiles/charset/A_utf8.txt")
    val objs = MyObject(_,1)).collect()
    for (x <- objs ) {

The class MyObject is also a very simple Class, which is only used to test the serialization
class MyObject  {
  var myStr : String = ""
  var myInt : Int = 0
  def this(inStr : String, inInt : Int) {
    this.myStr = inStr
    this.myInt = inInt
  def printMyObject {
    println("MyString is : "+myStr+"\tMyInt is : "+myInt)

But when I ran the application, it reported the following error:
java.lang.IllegalArgumentException: Class is not registered: dhao.test.Serialization.MyObject[]
Note: To register this class use: kryo.register(dhao.test.Serialization.MyObject[].class);
    at com.esotericsoftware.kryo.Kryo.getRegistration(
    at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(
    at com.esotericsoftware.kryo.Kryo.writeClass(
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:161)
    at org.apache.spark.executor.Executor$
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    at java.util.concurrent.ThreadPoolExecutor$

I don't understand what cause this problem. I have used the "conf.registerKryoClasses" to
register my class. Could anyone help me ? Thanks

By the way, the spark version is 1.3.0.
View raw message