beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-626) AvroCoder not deserializing correctly in Kryo
Date Sat, 29 Oct 2016 00:04:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617003#comment-15617003
] 

Luke Cwik commented on BEAM-626:
--------------------------------

This will only solve the short term problem that AvroCoder is not serializable via Kryo.

Users who write their own objects that rely on readResolve will still have the same problem
that your facing with AvroCoder. They will need to do additional work to get their objects
to work with the Spark runner.

We'll need an official schema / serialization story for many of the objects used such as Coder/DoFn/...
to be part of the Beam model for portability reasons but until then it seems worthwhile to
fix this.

> AvroCoder not deserializing correctly in Kryo
> ---------------------------------------------
>
>                 Key: BEAM-626
>                 URL: https://issues.apache.org/jira/browse/BEAM-626
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Aviem Zur
>            Assignee: Aviem Zur
>            Priority: Minor
>
> Unlike with Java serialization, when deserializing AvroCoder using Kryo, the resulting
AvroCoder is missing all of its transient fields.
> The reason it works with Java serialization is because of the usage of writeReplace and
readResolve, which Kryo does not adhere to.
> In ProtoCoder for example there are also unserializable members, the way it is solved
there is lazy initializing these members via their getters, so they are initialized in the
deserialized object on first call to the member.
> It seems AvroCoder is the only class in Beam to use writeReplace convention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message