kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Zhang <ning2008w...@gmail.com>
Subject Re: When using MM2, how should the consumer be switched from source to target?
Date Wed, 13 Jan 2021 02:17:19 GMT
Hello Aki,

Can you elaborate on "when there are some messages left in the original cluster" ? 

On 2021/01/12 13:01:40, Aki Yoshida <elakito@gmail.com> wrote: 
> Hi Ryanne,
> Thanks for the information regarding the manual translation approach.
> But for 2.7.0, I have a question. I thought this translation would
> happen implicitly.
> I saw test testOneWayReplicationWithAutoOffsetSync of
> MirrorConnectorsIntegrationTest that is supposed to test this
> behavior. But I have an impression that the consumers can't be
> migrated when there are some messages left in the original cluster. I
> posted this question to users@kafka a while ago but I didn't receive
> any responses.
> https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
> Could you or someone comment on this?
> Thanks.
> Regards, aki
> 
> El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ryannedolan@gmail.com>) escribió:
> >
> > Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> > from within your client code, or you can write a little command-line tool
> > to do it. I've done the latter for Cloudera's srm-control tool, as
> > documented here:
> > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
> >
> > Ryanne
> >
> > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <elakito@gmail.com> wrote:
> >
> > > I'm answering to my own mail and would like to hear if this assumption
> > > is correct.
> > > it looks like I need 2.7.0 to have the automatic translation.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > for earlier versions, there is no way to use the console client but
> > > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > > and re-consuming from the new topic.
> > > Is this correct?
> > >
> > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<elakito@gmail.com>)
> > > escribió:
> > > >
> > > > Hi,
> > > > I have a question regarding how to migrate a consumer when the
> > > > subscribing topic has been migrated to the target Kafka broker.
> > > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > > broker. When MM2 mirrors this topic using options
> > > > source.cluster.alias="", replication.policy.separator="", topic named
> > > > topic1 shows up at the target Kafka broker.
> > > >
> > > > When the consumer instance simply switches the bootstrap server to
> > > > start consuming from this topic1 at the target broker, the consumer
> > > > seems to start subscribing from the latest offset. I thought the
> > > > consumer offsets were also migrated to the target and the consumer
> > > > could simply start subscribing from the mirrored topic from the
> > > > continued offset. Did I miss some configuration parameters or does the
> > > > consumer need to perform some actions to be able to consume the
> > > > records seamlessly?
> > > >
> > > > I appreciate for your help. Thanks.
> > > >
> > > > regards, aki
> > >
> 

Mime
View raw message