flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3294) KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
Date Fri, 01 Jul 2016 07:20:11 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358546#comment-15358546

Robert Metzger commented on FLINK-3294:

Hi Jens,

the Kafka 0.9 consumer doesn't commit the offsets to Zookeeper. They are committed to the
Kafka broker.
Only Flink's Kafka 0.8 consumer commits to ZK. That's why the commit code is different in
the two implementations. The Kafka 0.9 consumer also should not have any ZK dependencies.

What you have to do is to introduce an interface "OffsetHandler" into the 0.8 fetcher that
has two implementations: A Zookeeper Offset handler (the current code) and a Kafka offset
handler (new code).
In the diff of the branch I posed above, you can see this structure already: https://github.com/apache/flink/compare/master...rmetzger:yangjun_fix

> KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
> -----------------------------------------------------------------------
>                 Key: FLINK-3294
>                 URL: https://issues.apache.org/jira/browse/FLINK-3294
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
> Currently, the 0.8 consumer for Kafka is committing the offsets manually into Zookeeper
so that users can track the lag using external tools.
> The 0.8 consumer has a pluggable design, and this component is easily pluggable.
> Since OffsetCommitRequest version=1 (supported in 0.8.2 or later), users can choose between
two offset commit modes:
> a) Let the broker commit into ZK (this is  what we are doing from the consumer
> b) Let the broker commit the offset into a special topic.
> By adding a different "OffsetHandler" backend, users can commit offsets from the brokers
(reducing the total number of ZK connections) or into the broker's offset topic.

This message was sent by Atlassian JIRA

View raw message