kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryanne Dolan <ryannedo...@gmail.com>
Subject Re: Consumer offsets with Kafka Mirror Maker
Date Mon, 24 Jun 2019 17:25:11 GMT
Mohit, PR-6295 and KIP-382 introduce MirorrMaker 2.0 which was designed to
support this operation.

In a nutshell, MM2 maintains a sparse offset sync stream while replicating
records between clusters. The offset syncs are used to translate consumer
offsets for periodic cross-cluster checkpoints. Leveraging these
checkpoints, you can resume processing in a new cluster via
kafka-consumer-groups.sh --reset-offsets, and you are guaranteed to never
skip records.

The RemoteClusterUtils.translateOffsets() method in the PR is a good place
to start. It will return the latest translated offsets for a particular
consumer group on a remote cluster. You can build tooling around this, e.g.
to migrate all your consumers at once.

https://github.com/apache/kafka/pull/6295
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0

Ryanne

On Mon, Jun 24, 2019 at 10:34 AM Mohit Kumar <mail@imohit.me> wrote:

> Is there anyway to copy consumers offset while moving data from one cluster
> to another cluster?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message