flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-4722) Consumer group concept not working properly with FlinkKafkaConsumer09
Date Mon, 03 Oct 2016 16:59:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542855#comment-15542855
] 

Tzu-Li (Gordon) Tai edited comment on FLINK-4722 at 10/3/16 4:58 PM:
---------------------------------------------------------------------

I'm not sure if I fully understand your use case, but you could consider this solution:
Use a single FlinkKafkaConsumer to read the topic, and key the input elements by the partition
id (you can do this by supplying your own {{KeyedDeserializationSchema}} to the consumer;
the deserialization schema exposes the partition of the read record, so you can use that to
key the data, ex. output {{Tuple(partitionId,matrixData)}} or your own POJO / case class directly
from the source).
Then, on the keyed stream, you can perform windows on each key (i.e., in your case, each matrix)
like you mentioned.

Does this help?


was (Author: tzulitai):
I'm not sure if I fully understand your use case, but you could consider this solution:
Use a single FlinkKafkaConsumer to read the topic, and key the input elements by the partition
id (you can do this by supplying your own {{KeyedDeserializationSchema}} to the consumer).
Then, on the keyed stream, you can perform windows on each key (i.e., in your case, each matrix)
like you mentioned.

Does this help?

> Consumer group concept not working properly with FlinkKafkaConsumer09  
> -----------------------------------------------------------------------
>
>                 Key: FLINK-4722
>                 URL: https://issues.apache.org/jira/browse/FLINK-4722
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.1.2
>            Reporter: Sudhanshu Sekhar Lenka
>
> When Kafka one Topic has 3 partition and 3 FlinkKafkaConsumer09 connected to that same
topic using "group.id" ,"myGroup" property . Still flink consumer get all data which are push
to each 3   partition . While it work properly with normal java consumer. each consumer get
specific data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message