kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Zhang <ning2008w...@gmail.com>
Subject Re: When using MM2, how should the consumer be switched from source to target?
Date Wed, 13 Jan 2021 17:19:29 GMT
hello Aki,

I guess overall you are right.

When the consumer have not finish consuming the latest message from primary and switch to
the backup,  the left messages will be still replicated from primary to backup by MM, so your
consumer will still be able to consume the left messages at backup, this is what your commit
illustrated.

I think the original test was correct as well, since it assumes no left message in primary
before switching consumer to backup.

If you are interested in contributing your commit, it would be great to submit it on https://github.com/apache/kafka/pull/9224,
as MM tests could be significantly refactored soon.

Thanks 

On 2021/01/13 08:14:06, Aki Yoshida <elakito@gmail.com> wrote: 
> Hi Ning,
> I mean by "when there are some messages left in the original cluster",
> when the consumer switches from the original cluster to the new
> cluster before entirely consuming the messages at the original
> cluster.
> This commit illustrates this situation.
> https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7
> 
> regards, aki
> 
> El mié, 13 ene 2021 a las 3:17, Ning Zhang (<ning2008wisc@gmail.com>) escribió:
> >
> > Hello Aki,
> >
> > Can you elaborate on "when there are some messages left in the original cluster"
?
> >
> > On 2021/01/12 13:01:40, Aki Yoshida <elakito@gmail.com> wrote:
> > > Hi Ryanne,
> > > Thanks for the information regarding the manual translation approach.
> > > But for 2.7.0, I have a question. I thought this translation would
> > > happen implicitly.
> > > I saw test testOneWayReplicationWithAutoOffsetSync of
> > > MirrorConnectorsIntegrationTest that is supposed to test this
> > > behavior. But I have an impression that the consumers can't be
> > > migrated when there are some messages left in the original cluster. I
> > > posted this question to users@kafka a while ago but I didn't receive
> > > any responses.
> > > https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
> > > Could you or someone comment on this?
> > > Thanks.
> > > Regards, aki
> > >
> > > El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ryannedolan@gmail.com>)
escribió:
> > > >
> > > > Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> > > > from within your client code, or you can write a little command-line tool
> > > > to do it. I've done the latter for Cloudera's srm-control tool, as
> > > > documented here:
> > > > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
> > > >
> > > > Ryanne
> > > >
> > > > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <elakito@gmail.com> wrote:
> > > >
> > > > > I'm answering to my own mail and would like to hear if this assumption
> > > > > is correct.
> > > > > it looks like I need 2.7.0 to have the automatic translation.
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > > > for earlier versions, there is no way to use the console client but
> > > > > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > > > > and re-consuming from the new topic.
> > > > > Is this correct?
> > > > >
> > > > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<elakito@gmail.com>)
> > > > > escribió:
> > > > > >
> > > > > > Hi,
> > > > > > I have a question regarding how to migrate a consumer when the
> > > > > > subscribing topic has been migrated to the target Kafka broker.
> > > > > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > > > > broker. When MM2 mirrors this topic using options
> > > > > > source.cluster.alias="", replication.policy.separator="", topic
named
> > > > > > topic1 shows up at the target Kafka broker.
> > > > > >
> > > > > > When the consumer instance simply switches the bootstrap server
to
> > > > > > start consuming from this topic1 at the target broker, the consumer
> > > > > > seems to start subscribing from the latest offset. I thought
the
> > > > > > consumer offsets were also migrated to the target and the consumer
> > > > > > could simply start subscribing from the mirrored topic from
the
> > > > > > continued offset. Did I miss some configuration parameters or
does the
> > > > > > consumer need to perform some actions to be able to consume
the
> > > > > > records seamlessly?
> > > > > >
> > > > > > I appreciate for your help. Thanks.
> > > > > >
> > > > > > regards, aki
> > > > >
> > >
> 

Mime
View raw message