kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.INVALID>
Subject Re: MirrorMaker2 ordering guarantees
Date Tue, 28 Apr 2020 13:27:52 GMT
Hey,

so I was just about to set `max.in.flight.requests.per.connection=1` when I
found in the logs that it is already set for the "destination" producers :)
Cool.

Duplicates are fine for me, we have dedup checks or immutable operations on
the consumers.

Have a nice day!
Peter

On Mon, 27 Apr 2020 at 19:45, Ryanne Dolan <ryannedolan@gmail.com> wrote:

> > Could the `max.in.flight.requests.per.connection=1` parameter help to
> > prevent the "slightly out-of-order records"?
>
> Yes that helps, but dupes are still possible when MM2 restarts or
> rebalances, since it will restart at the latest commit. If you are
> replicating something like CDC or changelogs, then dupes might be fine
> (downstream state will be eventually consistent). That's a common pattern
> with MM1 as well.
>
> Ryanne
>
> On Mon, Apr 27, 2020 at 4:47 AM Péter Sinóros-Szabó
> <peter.sinoros-szabo@transferwise.com.invalid> wrote:
>
> > Hey Ryanne,
> >
> > Is there any documentation where I can read more about this "slightly
> > out-of-order records"?
> > It would help very much to see how we can use MM2 in our systems.
> >
> > Thanks,
> > Peter
> >
> > On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó <
> > peter.sinoros-szabo@transferwise.com> wrote:
> >
> > > Hey Ryanna,
> > >
> > > Could the `max.in.flight.requests.per.connection=1` parameter help to
> > > prevent the "slightly out-of-order records"?
> > > Or is there any workaround for that? Duplicates are fine for me, but
> I'd
> > > like to have the same order of messages too.
> > > Can you please add some more detail about why those "slightly
> > out-of-order
> > > records" may happen?
> > >
> > > Thanks,
> > > Peter
> > >
> > > On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ryannedolan@gmail.com>
> > wrote:
> > >
> > >> Hey Peter, Connect will need to support transactions before we can
> > >> guarantee the order of records in remote topics. We can guarantee that
> > no
> > >> records are dropped or skipped, even during consumer
> failover/migration
> > >> etc, but we can still have duplicates and slightly out-of-order
> records
> > in
> > >> the downstream remote topics, for now.
> > >>
> > >> Ryanne
> > >>
> > >> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó
> > >> <peter.sinoros-szabo@transferwise.com.invalid> wrote:
> > >>
> > >> > Hey,
> > >> >
> > >> > so KIP-382 mentions that:
> > >> > "Partitioning and order of records is preserved between source and
> > >> remote
> > >> > topics."
> > >> > is the ordering of messages (I guess only in a partition) something
> > >> that is
> > >> > actually implemented in 2.4 (or in 2.5)?
> > >> >
> > >> > Or do I need to set `max.in.flight.requests.per.connection=1` ?
> > >> >
> > >> > Thanks,
> > >> > Peter
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message