beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-626) AvroCoder not deserializing correctly in Kryo
Date Sun, 30 Oct 2016 07:55:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15619517#comment-15619517
] 

Amit Sela commented on BEAM-626:
--------------------------------

[~lcwik] this is not accurate.
Users who write their own *Coders* will have this problem. The Spark runner serializes user
data using Coders (and where it fails to do so, it's a bug), so this won't be an issue.
BTW, user data serialization with Kryo is common, though most runners (Spark too) serialize
with Coders per/post shuffle.

> AvroCoder not deserializing correctly in Kryo
> ---------------------------------------------
>
>                 Key: BEAM-626
>                 URL: https://issues.apache.org/jira/browse/BEAM-626
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Aviem Zur
>            Assignee: Aviem Zur
>            Priority: Minor
>
> Unlike with Java serialization, when deserializing AvroCoder using Kryo, the resulting
AvroCoder is missing all of its transient fields.
> The reason it works with Java serialization is because of the usage of writeReplace and
readResolve, which Kryo does not adhere to.
> In ProtoCoder for example there are also unserializable members, the way it is solved
there is lazy initializing these members via their getters, so they are initialized in the
deserialized object on first call to the member.
> It seems AvroCoder is the only class in Beam to use writeReplace convention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message