kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aki Yoshida <elak...@gmail.com>
Subject Re: When using MM2, how should the consumer be switched from source to target?
Date Wed, 13 Jan 2021 08:14:06 GMT
Hi Ning,
I mean by "when there are some messages left in the original cluster",
when the consumer switches from the original cluster to the new
cluster before entirely consuming the messages at the original
cluster.
This commit illustrates this situation.
https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7

regards, aki

El mié, 13 ene 2021 a las 3:17, Ning Zhang (<ning2008wisc@gmail.com>) escribió:
>
> Hello Aki,
>
> Can you elaborate on "when there are some messages left in the original cluster" ?
>
> On 2021/01/12 13:01:40, Aki Yoshida <elakito@gmail.com> wrote:
> > Hi Ryanne,
> > Thanks for the information regarding the manual translation approach.
> > But for 2.7.0, I have a question. I thought this translation would
> > happen implicitly.
> > I saw test testOneWayReplicationWithAutoOffsetSync of
> > MirrorConnectorsIntegrationTest that is supposed to test this
> > behavior. But I have an impression that the consumers can't be
> > migrated when there are some messages left in the original cluster. I
> > posted this question to users@kafka a while ago but I didn't receive
> > any responses.
> > https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
> > Could you or someone comment on this?
> > Thanks.
> > Regards, aki
> >
> > El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ryannedolan@gmail.com>) escribió:
> > >
> > > Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> > > from within your client code, or you can write a little command-line tool
> > > to do it. I've done the latter for Cloudera's srm-control tool, as
> > > documented here:
> > > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
> > >
> > > Ryanne
> > >
> > > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <elakito@gmail.com> wrote:
> > >
> > > > I'm answering to my own mail and would like to hear if this assumption
> > > > is correct.
> > > > it looks like I need 2.7.0 to have the automatic translation.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > > for earlier versions, there is no way to use the console client but
> > > > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > > > and re-consuming from the new topic.
> > > > Is this correct?
> > > >
> > > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<elakito@gmail.com>)
> > > > escribió:
> > > > >
> > > > > Hi,
> > > > > I have a question regarding how to migrate a consumer when the
> > > > > subscribing topic has been migrated to the target Kafka broker.
> > > > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > > > broker. When MM2 mirrors this topic using options
> > > > > source.cluster.alias="", replication.policy.separator="", topic named
> > > > > topic1 shows up at the target Kafka broker.
> > > > >
> > > > > When the consumer instance simply switches the bootstrap server to
> > > > > start consuming from this topic1 at the target broker, the consumer
> > > > > seems to start subscribing from the latest offset. I thought the
> > > > > consumer offsets were also migrated to the target and the consumer
> > > > > could simply start subscribing from the mirrored topic from the
> > > > > continued offset. Did I miss some configuration parameters or does
the
> > > > > consumer need to perform some actions to be able to consume the
> > > > > records seamlessly?
> > > > >
> > > > > I appreciate for your help. Thanks.
> > > > >
> > > > > regards, aki
> > > >
> >

Mime
View raw message