spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 颜发才(Yan Facai) <yaf...@gmail.com>
Subject Re: Spark 2.0.2, KyroSerializer, double[] is not registered.
Date Sun, 08 Jan 2017 13:28:39 GMT
Hi, Owen,
it is fixed after registering manually:
conf.registerKryoClasses(Array(Class.forName("[D")))


I believe that Kyro (latest version) have supported double[] :
addDefaultSerializer(double[].class, DoubleArraySerializer.class);
<https://github.com/EsotericSoftware/kryo/blob/e2fe2a4c339f48120e71ae61967f76ddfcf4ba3a/src/com/esotericsoftware/kryo/Kryo.java#L197>

Why does it break in spark?



On Sun, Jan 8, 2017 at 6:03 PM, Sean Owen <sowen@cloudera.com> wrote:

> Double[] is not of the same class as double[]. Kryo should already know
> how to serialize double[], but I doubt Double[] is registered.
>
> The error does seem to clearly indicate double[] though. That surprises
> me.  Can you try manually registering it to see if that fixes it?
> But then I'm not sure why tests wouldn't catch this.
>
> On Sun, Jan 8, 2017 at 7:30 AM smartzjp <zjp_jdev@163.com> wrote:
>
>> You can have a try the following code.
>>
>> ObjectArraySerializer serializer = new ObjectArraySerializer(kryo, Double
>> [].class);
>> kryo.register(Double[].class, serializer);
>>
>>
>> -----------
>>
>> Hi, all.
>> I enable kyro in spark with spark-defaults.conf:
>>      spark.serializer                 org.apache.spark.serializer.
>> KryoSerializer
>>      spark.kryo.registrationRequired  true
>>
>> A KryoException is raised when a logistic regression algorithm is running:
>>      Note: To register this class use: kryo.register(double[].class);
>>      Serialization trace:
>>      currL1 (org.apache.spark.mllib.stat.MultivariateOnlineSummarizer)
>>             at com.esotericsoftware.kryo.serializers.FieldSerializer$
>> ObjectField.write(FieldSerializer.java:585)
>>             at com.esotericsoftware.kryo.serializers.FieldSerializer.
>> write(FieldSerializer.java:213)
>>             at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
>> java:568)
>>             at com.twitter.chill.Tuple2Serializer.write(
>> TupleSerializers.scala:36)
>>             at com.twitter.chill.Tuple2Serializer.write(
>> TupleSerializers.scala:33)
>>            at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.
>> java:568)
>>
>> My question is:
>> Doesn't double[].class be supported by default?
>>
>> Thanks.
>>
>>

Mime
View raw message