kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Zhang <ning2008w...@gmail.com>
Subject Re: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0
Date Sun, 06 Dec 2020 22:15:51 GMT
Hi Amit, I guess you may need to override the actual consumer group config (probably not consumer.group.id)
that is used in Kafka Connect 

On 2020/11/26 06:47:11, <Amit.SRIVASTAV@cognizant.com> wrote: 
> Hi All,
> 
> We are currently trying to migrate Confluent replicator to Apache Open Source Mirror
Maker v2.0. We are facing an issue where the messages which are already replicated by replicator
is getting replicated again when the mirror maker is started on the same topic. This should
not happen as messages are getting duplicated at the target cluster. Here are more details:
> 
> 
> 1.       RCA: replicator assign a consumer group for replicating messages. This consumer
group maintains the offset of the source topic. But we are not able to assign same consumer
group to the Consumer config in mirror maker 2.
> 
> 2.       Mirror Maker 1.0 : working as same consumer group can be assigned in consumer.properties
file and the messages are picked right after where replicator was stopped.
> 
> 3.       Tried running and configuring source.cluster.consumer.group.id in mirror maker
2.0 in all available options (in cluster mode, in connect-standalone and connect-distributed
mode) but mirror maker 2.0 is assigning consumer group id as null while replicating messages.
> 
> 
> Any pointers if anyone has done same and tried to maintain the same offset with mirror
maker 2.0.
> 
> Thanks and regards,
> Amit
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s)
and may contain confidential and privileged information. If you are not the intended recipient(s),
please reply to the sender and destroy all copies of the original message. Any unauthorized
review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or
any action taken in reliance on the contents of this e-mail is strictly prohibited and may
be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications
sent to and from Cognizant e-mail addresses may be monitored.
> 

Mime
View raw message