flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7420) Move all Avro code to flink-avro
Date Wed, 01 Nov 2017 08:01:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233758#comment-16233758
] 

ASF GitHub Bot commented on FLINK-7420:
---------------------------------------

GitHub user aljoscha opened a pull request:

    https://github.com/apache/flink/pull/4931

    [FLINK-7420] Move all Avro code to flink-avro

    This hides interaction with Avro begin the interface `AvroUtils` that has two implementations:
1) default Avro utils, which are used when no Avro is present and which throws exceptions
in case Avro is required. 2) proper Avro utils which are dynamically loaded when the `flink-avro`
module is "in the classpath" which does proper Avro things.
    
    R: @StephanEwen 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink FLINK-7420

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4931.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4931
    
----
commit b74f84ee3d5795e30c92a2e7393fc25309c450e3
Author: twalthr <twalthr@apache.org>
Date:   2017-08-16T10:17:00Z

    [FLINK-7420] [core] Move all Avro code to flink-avro

commit 3215a4b939f536b4a1130cad9e5106867c071789
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-10-25T15:38:24Z

    [FLINK-7420] Replace GenericData.Array by dummy when reading TypeSerializers
    
    This also adds a new test that verifies that we correctly register
    Avro Serializers when they are present and modifies an existing test to
    verify that we correctly register dummy classes.

commit 94f19afcb65b7ef0d200bb3c4d0d82b8422ba905
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-10-26T12:56:09Z

    [FLINK-7420] Add Avro test-jar depdendency in Kafka modules

commit 1d73f296667d909ce0506ceeb722112fed978af3
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-10-30T09:19:56Z

    [FLINK-7420] Fix TwitterExample.scala
    
    It seems this has a transitive dependency on Jackson, which slightly
    changed with the Avro reworking.

commit a7289e06641b3303611bf5a1a8a2bf4ef56ac994
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-10-30T14:02:18Z

    [FLINK-7420] Abstract all Avro interaction behind AvroUtils
    
    Before, we would try and dynamicall load Avro-related classes in several
    places. Now, we only reflectively instantiate the right AvroUtils and
    all other operations are methods on this.
    
    The default AvroUtils throw exceptions with a helpful message for most
    operations.

----


> Move all Avro code to flink-avro
> --------------------------------
>
>                 Key: FLINK-7420
>                 URL: https://issues.apache.org/jira/browse/FLINK-7420
>             Project: Flink
>          Issue Type: Improvement
>          Components: Build System
>            Reporter: Stephan Ewen
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: 1.4.0
>
>
> *Problem*
> Currently, the {{flink-avro}} project is a shell with some tests and mostly duplicate
and dead code. The classes that use Avro are distributed quite wildly through the code base,
and introduce multiple direct dependencies on Avro in a messy way.
> That way, we cannot create a proper fat Avro dependency in which we shade Jackson away.
> Also, we expose Avro as a direct and hard dependency on many Flink modules, while it
should be a dependency that users that use Avro types selectively add.
> *Suggested Changes*
> We should move all Avro related classes to {{flink-avro}}, and give {{flink-avro}} a
dependency on {{flink-core}} and {{flink-streaming-java}}.
>   - {{AvroTypeInfo}}
>   - {{AvroSerializer}}
>   - {{AvroRowSerializationSchema}}
>   - {{AvroRowDeserializationSchema}}
> To be able to move the the avro serialization code from {{flink-ore}} to {{flink-avro}},
we need to load the {{AvroTypeInformation}} reflectively, similar to how we load the {{WritableTypeInfo}}
for Hadoop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message