kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunny Lohani <sunny.loh...@gmail.com>
Subject Re: Kafka Mirror Maker 2: RemoteClusterUtils.translateOffsets() returning empty map
Date Tue, 04 Aug 2020 15:26:59 GMT
Hi Ryanne,

First of all, thanks for a quick revert. Actually, I have a consumer
application consuming messages in cluster A and then I failover the
consumer from A to cluster B. Also, the consumer is subscribed to both
local and remote topics. Let's say the local topic name in both the
clusters A and B is test-topic. Here is a sequence of steps that I am
following before and after the failover:

1. Consumer application subscribes to topics test-topic and B.test-topic in
cluster A with group.id "test-consumer"
2. It consumes some messages from both the topics.
3. Now, we stop the consumer application and restart it pointing to cluster
B, subscribed to topics test-topic and A.test-topic, with group.id
"test-consumer". I use the RemoteClusterUtils.translateOffsets() here.
4. The method returns an empty map as well as the checkpoints topic in both
the clusters is empty.

Let me know if you see anything wrong here.

Thanks,
Sunny


On Tue, Aug 4, 2020 at 8:26 PM Ryanne Dolan <ryannedolan@gmail.com> wrote:

> Sunny, is it possible there are no consumer groups? There will be no
> checkpoints, and thus nothing to use for offset translation, if there are
> no upstream consumer groups.
>
> Ryanne
>
> On Tue, Aug 4, 2020, 9:28 AM Sunny Lohani <sunny.lohani@gmail.com> wrote:
>
> > Hi,
> >
> > I have 2 data centers, each having single node Zookeeper and Kafka
> cluster.
> > I have a topic (single partition) in both the data center kafka
> clusters. I
> > am using MM 2.0 as a dedicated cluster for bi-directional replication of
> > the topic as well as using RemoteClusterUtils.translateOffsets() in my
> > application for offset translation during failover. But the method is
> > returning an empty map due to which the consumer is not resuming from
> > proper offsets for local/remote topics.
> >
> > When I investigated further, I found that the checkpoint
> > topics A.checkpoints.internal and B.checkpoints.internal in respective
> > clusters do not have any kafka message. I don't see any errors in the
> > mirror maker console logs. I searched everywhere on the internet but
> could
> > not get any help. Below is the mm.properties:
> >
> > clusters = A, B
> > A.bootstrap.servers = 10.34.45.113:19092
> > B.bootstrap.servers = 10.34.45.113:29092
> >
> > A->B.enabled = true
> > A->B.topics = .*
> > B->A.enabled = true
> > B->A.topics = .*
> >
> > # Setting replication factor of newly created remote topics
> > replication.factor=1
> >
> > checkpoints.topic.replication.factor=1
> > heartbeats.topic.replication.factor=1
> > offset-syncs.topic.replication.factor=1
> >
> > offset.storage.replication.factor=1
> > status.storage.replication.factor=1
> > config.storage.replication.factor=1
> >
> > sync.topic.acls.enabled = false
> >
> > emit.checkpoints.enabled = true
> > emit.checkpoints.interval.seconds = 5
> > ----
> >
> > Need help on this urgently. Thanks in advance.
> >
> > Thanks & Regards,
> > Sunny Kumar Lohani,
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message