flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chaiyongqiang (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-13052) Supporting multi-topic when using kafkaTableSourceSinkFactoryBase.createStreamTableSource
Date Tue, 12 Nov 2019 10:09:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972243#comment-16972243
] 

chaiyongqiang edited comment on FLINK-13052 at 11/12/19 10:08 AM:
------------------------------------------------------------------

There're some limits when we support multi source topics in kafkaTableSourceSinkFactoryBase.
It's like :

When we assign the startup mode as specific-offsets, it's difficult to assign the start-offset
for each topic partition because of the way to set the connector configs.  As the following
codes show, when we assign several topics like MyTable, MyTable1,MyTable2 , it's difficult
to distinguish the partition and offset belong to wich topic. 

{code:java}
final Map<String, String> props2 = new HashMap<>();
		props2.put("connector.property-version", "1");
		props2.put("connector.type", "kafka");
		props2.put("connector.version", "0.11");
		props2.put("connector.topic", "MyTable");
		props2.put("connector.startup-mode", "specific-offsets");
		props2.put("connector.specific-offsets.0.partition", "0");
		props2.put("connector.specific-offsets.0.offset", "42");
		props2.put("connector.specific-offsets.1.partition", "1");
{code}

But fortunately´╝î setting a fixed topic-partition start offset when we start an app with
table API is rare. 
My solution is supporting multi-topic for other kind of start-up mode like GROUP_OFFSETS,
EARLIEST,LATEST,  for SPECIFIC_OFFSETS mode  when using Table API like kafkaTableSourceSinkFactoryBase
with several topics we use the GROUP_OFFSETS instead.  
Does someone have a better idea ? Please let me know. Many thanks


was (Author: chaiyq):
There're some limits when we support multi source topics in kafkaTableSourceSinkFactoryBase.
It's like :

When we assign the startup mode as specific-offsets, it's difficult to assign the start-offset
for each topic partition because of the way to set the connector configs.  As the following
codes show, when we assign several topics like MyTable, MyTable1,MyTable2 , it's difficult
to distinguish the partition and offset belong to wich topic. 

{code:java}
final Map<String, String> props2 = new HashMap<>();
		props2.put("connector.property-version", "1");
		props2.put("connector.type", "kafka");
		props2.put("connector.version", "0.11");
		props2.put("connector.topic", "MyTable");
		props2.put("connector.startup-mode", "specific-offsets");
		props2.put("connector.specific-offsets.0.partition", "0");
		props2.put("connector.specific-offsets.0.offset", "42");
		props2.put("connector.specific-offsets.1.partition", "1");
{code}

But fortunately´╝î setting a fixed topic-partition start offset when we start an app with
table API is rare. For now,  i suggest ignoring this situation.  My solution is supporting
multi-topic for other kind of start-up mode like GROUP_OFFSETS, EARLIEST,LATEST,  for SPECIFIC_OFFSETS
mode  when using Table API like kafkaTableSourceSinkFactoryBase with several topics we use
the GROUP_OFFSETS instead. 

> Supporting multi-topic when using kafkaTableSourceSinkFactoryBase.createStreamTableSource
> -----------------------------------------------------------------------------------------
>
>                 Key: FLINK-13052
>                 URL: https://issues.apache.org/jira/browse/FLINK-13052
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka, Table SQL / API
>    Affects Versions: 1.8.0
>            Reporter: chaiyongqiang
>            Priority: Critical
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message