flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Wang (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-14729) Multi-topics consuming from KafkaTableSource
Date Tue, 26 Nov 2019 15:54:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982624#comment-16982624

Matt Wang commented on FLINK-14729:

Good feature, just modify the KafkaTableSourceBase and  KafkaTableSource can support this

FlinkKafkaConsumer has already supported to consume  multiple topics at the same time.

> Multi-topics consuming from KafkaTableSource
> --------------------------------------------
>                 Key: FLINK-14729
>                 URL: https://issues.apache.org/jira/browse/FLINK-14729
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Kafka
>            Reporter: Leo Zhang
>            Priority: Major
>              Labels: features
> Hi, all. I propose a new functionality of KafkaTableSource which can consume multiple
topics at the same time. 
> *Design plan*
>  * Add a new constructor in KafkaTableSource which accepts topics with List type as one
>  * Modify the existed one which only accepts one topic as string type to call the proposed
one to finish the instantiation. That is to say, wrap this topic in a list and pass it to
the multi-topics-consuming constructor.
>  * Modify the overridden method createKafkaConsumer in KafkaTableSource to pass topics
as List instead of String.
>  * Replace the field topic with topics as List type in  KafkaTableSourceBase and modify
every place using topic with topics. So we just need to modify the constructor KafkaTableSourceBase,
method getDataStream, and equals and hashCode.
> *Test plan*
> There is less to do as KafkaTableSource is based on FlinkKafkaConsumer which already
supports consuming multiple topics and is tested well. Of course, we can easily add further
more tests if needed.
> So what's your opinion?

This message was sent by Atlassian Jira

View raw message