flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8537) Add a Kafka table source factory with Avro format support
Date Thu, 01 Mar 2018 12:05:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381893#comment-16381893
] 

ASF GitHub Bot commented on FLINK-8537:
---------------------------------------

GitHub user xccui opened a pull request:

    https://github.com/apache/flink/pull/5610

    [FLINK-8537][table]Add a Kafka table source factory with Avro format support

    ## What is the purpose of the change
    
    This PR adds the Kafka table source factory with Avro format.
    
    ## Brief change log
    
    ### Adding
      - Add an `Avro` format descriptor and the corresponding validator.
      - Add a `KafkaAvroTableSourceFactory` and its version specific implementations.
      - Add a `KafkaTableSourceFactory` and make the json/avro factory extend it.
      - Add `equals()` and `hashCode()` methods to the timestamp/watermark extractors for
testing.
    
    ### Updating
      - Change the schema configuration logic for `KafkaJsonTableSourceFactory`.
      - Add a `withFormat` flag to `KafkaJsonTableSourceFactory` test cases and replace the
proctime attributes with the rowtime ones.
      - Move the `convertToRowTypeInformation()` method to a new helper class `AvroTypeInfoConverter`.
    
    ### Dependency
      - Make `flink-avro` depend on `flink-table` and `flink-streaming-scala`.
      - Remove the `flink-avro` dependency from `flink-table` and move the only effcted test
`AvroTypesITCase` to `flink-avro`.
    
    ## Verifying this change
    
    This change can be verified by the test cases in `AvroTest` and `KafkaAvroTableSourceFactoryTestBase`.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (yes)
      - If yes, how is the feature documented? (JavaDocs)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xccui/flink FLINK-8537

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5610.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5610
    
----
commit 99387fcd699267bbf5f778e5389060eae5420a4e
Author: Xingcan Cui <xingcanc@...>
Date:   2018-03-01T11:21:05Z

    [FLINK-8537][table]Add a Kafka table source factory with Avro format support

----


> Add a Kafka table source factory with Avro format support
> ---------------------------------------------------------
>
>                 Key: FLINK-8537
>                 URL: https://issues.apache.org/jira/browse/FLINK-8537
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table API &amp; SQL
>            Reporter: Timo Walther
>            Assignee: Xingcan Cui
>            Priority: Major
>
> Similar to {{CSVTableSourceFactory}} a Kafka table source factory should be added. This
issue includes creating a {{Avro}} descriptor with validation that can be used for other connectors
as well. It is up for discussion if we want to split the KafkaAvroTableSource into connector
and format such that we can reuse the format for other table sources as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message