spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Lists <lists.a...@gmail.com>
Subject Re: ClassNotFoundException when registering classes with Kryo
Date Mon, 02 Feb 2015 07:11:12 GMT
Thanks for the notification!

For now, I'll use the Kryo serializer without registering classes until the
bug fix has been merged into the next version of Spark (I guess that will
be 1.3, right?).

arun


On Sun, Feb 1, 2015 at 10:58 PM, Shixiong Zhu <zsxwing@gmail.com> wrote:

> It's a bug that has been fixed in
> https://github.com/apache/spark/pull/4258 but not yet been merged.
>
> Best Regards,
> Shixiong Zhu
>
> 2015-02-02 10:08 GMT+08:00 Arun Lists <lists.arun@gmail.com>:
>
>> Here is the relevant snippet of code in my main program:
>>
>> ===========================================
>>
>> sparkConf.set("spark.serializer",
>>   "org.apache.spark.serializer.KryoSerializer")
>> sparkConf.set("spark.kryo.registrationRequired", "true")
>> val summaryDataClass = classOf[SummaryData]
>> val summaryViewClass = classOf[SummaryView]
>> sparkConf.registerKryoClasses(Array(
>>
>>   summaryDataClass, summaryViewClass))
>>
>> ===========================================
>>
>> I get the following error:
>>
>> Exception in thread "main" java.lang.reflect.InvocationTargetException
>> ...
>>
>> Caused by: org.apache.spark.SparkException: Failed to load class to
>> register with Kryo
>> ...
>>
>> Caused by: java.lang.ClassNotFoundException:
>> com.dtex.analysis.transform.SummaryData
>>
>>
>> Note that the class in question SummaryData is in the same package as the
>> main program and hence in the same jar.
>>
>> What do I need to do to make this work?
>>
>> Thanks,
>> arun
>>
>>
>>
>

Mime
View raw message