flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Serialization and kryo
Date Sun, 16 Aug 2015 15:01:30 GMT
Hi!

The issue you refer to is: Flink automatically registers types it
encounters at Kryo, if the types are not handled by Flink itself. You can
turn that off in any program by
"ExecutionEnvionment.getConfig().disableAutoTypeRegistration()".

I am not sure that causes your problem, as this makes Kryo use tags rather
then class names for the types.

It may simply be that Kryo does not work with that data type out of the
box. It my try to treat the guava immutable map like a regular mp, causing
your exception. If that is really the problem, you can register a specific
serializer via "ExecuionEnvironment.registerTypeWithKryoSerializer(...)."

Let us know what is the problem in the end.

Greetings,
Stephan


On Sun, Aug 16, 2015 at 4:54 PM, jay vyas <jayunit100.apache@gmail.com>
wrote:

>
>    1. Just wanted to updated.. possibly I'm hitting ? FLINK-1417
>    Automatically register nested types at Kryo
>    <https://issues.apache.org/jira/browse/FLINK-1417> ...
>
>
> On Sun, Aug 16, 2015 at 10:38 AM, jay vyas <jayunit100.apache@gmail.com>
> wrote:
>
>> Hi flink.
>>
>> I just ran into the following serialization error in BigPetStore Flink.
>> It appears to be that someone is trying to add elements to a map during
>> serialization.   I doubt this is a bug in flink because it would have
>> surely been caught early on.
>>
>> Some possible explanations ...
>>
>> 1 - Probably its a conflict caused by my application's dependencies ?
>>
>> 2 - Or maybe just something about the way the mini-cluster sets up its
>> dependencies?.
>>
>> 3 - Or... Is this related to a guava transitive dependency conflict?
>>
>> DETAILS
>>
>> Our data generator library has these dependencies... Just FYI...
>> [INFO] \-
>> com.github.rnowling.bigpetstore:bigpetstore-data-generator:jar:0.2.1:compile
>> [INFO]    +- com.google.guava:guava:jar:18.0:compile
>> [INFO]    \- com.google.code.gson:gson:jar:2.3:compile
>>
>>
>> And, heres the exception:
>>
>> Caused by: com.esotericsoftware.kryo.KryoException:
>> java.lang.UnsupportedOperationException
>> Serialization trace:
>> fields (com.github.rnowling.bps.datagenerator.datamodels.Product)
>> products (com.github.rnowling.bps.datagenerator.datamodels.Transaction)
>> at
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>> at
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>> at
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>> at
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>> at
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>> at
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>
>> Caused by: java.lang.UnsupportedOperationException
>> at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:326)
>> at
>> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:144)
>> at
>> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>> at
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>> ... 18 more
>>
>
>
>
> --
> jay vyas
>

Mime
View raw message