kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aki Yoshida <elak...@gmail.com>
Subject Re: When using MM2, how should the consumer be switched from source to target?
Date Tue, 12 Jan 2021 13:01:40 GMT
Hi Ryanne,
Thanks for the information regarding the manual translation approach.
But for 2.7.0, I have a question. I thought this translation would
happen implicitly.
I saw test testOneWayReplicationWithAutoOffsetSync of
MirrorConnectorsIntegrationTest that is supposed to test this
behavior. But I have an impression that the consumers can't be
migrated when there are some messages left in the original cluster. I
posted this question to users@kafka a while ago but I didn't receive
any responses.
https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
Could you or someone comment on this?
Thanks.
Regards, aki

El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ryannedolan@gmail.com>) escribió:
>
> Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> from within your client code, or you can write a little command-line tool
> to do it. I've done the latter for Cloudera's srm-control tool, as
> documented here:
> https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
>
> Ryanne
>
> On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <elakito@gmail.com> wrote:
>
> > I'm answering to my own mail and would like to hear if this assumption
> > is correct.
> > it looks like I need 2.7.0 to have the automatic translation.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > for earlier versions, there is no way to use the console client but
> > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > and re-consuming from the new topic.
> > Is this correct?
> >
> > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<elakito@gmail.com>)
> > escribió:
> > >
> > > Hi,
> > > I have a question regarding how to migrate a consumer when the
> > > subscribing topic has been migrated to the target Kafka broker.
> > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > broker. When MM2 mirrors this topic using options
> > > source.cluster.alias="", replication.policy.separator="", topic named
> > > topic1 shows up at the target Kafka broker.
> > >
> > > When the consumer instance simply switches the bootstrap server to
> > > start consuming from this topic1 at the target broker, the consumer
> > > seems to start subscribing from the latest offset. I thought the
> > > consumer offsets were also migrated to the target and the consumer
> > > could simply start subscribing from the mirrored topic from the
> > > continued offset. Did I miss some configuration parameters or does the
> > > consumer need to perform some actions to be able to consume the
> > > records seamlessly?
> > >
> > > I appreciate for your help. Thanks.
> > >
> > > regards, aki
> >

Mime
View raw message