kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillaume Arnaud <garnau...@gmail.com>
Subject Re: Compare an original topic with a mirrored topic
Date Mon, 04 Nov 2019 23:52:43 GMT
Definetely it's a good start but currently we're building our
disaster/recovery processes and some topic contents are really important.
So in a paranoiac mode we want to some guarantees that messages are not
corrupted. Are there some internals in storage mechanisms during the send
and the replication which mitigate this kind of problem (I don't mean
acknowledgement processes but for data themself)?

In the worst case, we could never send a message and duplicate an other and
so have the same watermark. I guess that by design MirrorMaker v2 can't
have this use case but we are paranoiac :)

Le lun. 4 nov. 2019 à 23:58, M. Manna <manmedia@gmail.com> a écrit :

> How about High watermark check ?
>
> Since consumers consume based on HWM, presence of the same HWM should be a
> good checkpoint, no?
>
> Regards,
>
> On Mon, 4 Nov 2019 at 22:53, Guillaume Arnaud <garnaud25@gmail.com> wrote:
>
> > Hi,
> >
> > I would like to compare the messages of an original topic with a mirrored
> > topic in an other cluster to be sure that the content is the same.
> >
> > I see that checksum method in KafkaConsumer is deprecated:
> >
> >
> https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> >
> > What is the best way to do that ?
> >
> > Thank you,
> >
> > Guillaume
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message