flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StephanEwen <...@git.apache.org>
Subject [GitHub] flink issue #4943: [FLINK-6022] [avro] Use Avro to serialize Avro in flight ...
Date Mon, 06 Nov 2017 13:59:07 GMT
Github user StephanEwen commented on the issue:

    I updated this PR with the following proposed solution:
      1. Avro is always part of the user code space, and hence will be loaded into the user
code classloader. This solves multiple problems:
         - It allows users to use a different Avro version compared to the version in a spoiled
classpath (for example when Hadoop depenencies pull in another Avro version).
         - It means that Avro class and schema caching is done per user code classloader,
not JVM wide. That prevents "X cannot be cast to X" exceptions.
      2. By default, a new pure Avro serializer is used for Avro types (in flight and in state).
Since Avro class format changed, having compatibility activated by default is not possible
      3. If a user wants to recover an old savepoint that happend to have Avro types serialized
as PoJos, they need to do the following;
         - Set the Avro version to the same version that they had when creating the savepoint.
Since Avro now lives in user code space, that is possible without changing/rebuilding Flink.
         - Pass a Flag to the `AvroTypeInfo` to use a backwards compatible Serializer: `new
AvroTypeInfo(myType, true)`.


View raw message