kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergi Vladykin <sergi.vlady...@gmail.com>
Subject Re: Transactional Producer
Date Mon, 25 Nov 2019 06:50:34 GMT
Thanks a lot for your help!

Another question about ordering and visibility.

Lets say we have two transactional producers with different transactional
ids. They both publish records to the same partition like this:

*thread1: startTx*
*thread1: record A*
*thread2: startTx*
*thread2: record X*
*thread1: record B*
*thread1: commit*
*thread2: record Y*
*thread2: commit*

What order can I expect for these published messages?
Looks like it should be possible to get interleaved AXBY order (if the
records were not batched together).
But what if thread2 hangs for a long time right before the commit and
thread1 successfully commits?
We should get all the records in the order of their offsets, thus we will
be able to consume A and will not be able
to consume B until X is either committed or aborted?
Is my understanding right?

The same will happen when we have one transactional and one
non-transactional producer publishing to the same partition?

Sergi

вс, 24 нояб. 2019 г. в 21:12, Jonathan Santilli <jonathansantilli@gmail.com
>:

> Hello Sergi,
>
> 1. Is it OK to mix transactional and non-transactional approach with a
> single KafkaProducer instance?
> - This is not possible, a transactional producer can not send data outside
> a transaction.
>
> I mean sometimes I want to publish multiple messages transactionally, but
> oftentimes just a single message.
> Starting a transaction for publishing a single message looks inefficient.
> What is the recommend approach here?
> - Try to batch the records, if possible, otherwise, you need to begging and
> commit the transaction, even for a single record.
>
> 2. If I publish multiple messages to multiple partitions in a single
> transaction is it guaranteed to be all or nothing published?
> - Yes, this is the power of the transactions, all or nothing.
>
> Is it possible to end up with only half of the messages published to half
> of partitions in some failure scenario?
> - No, this is not possible if you are using correctly a transaction.
>
> Please, take a look at this simple gist with diff scenarios of a
> KafkaProducer, hope this help:
> https://gist.github.com/jonathansantilli/3b69ebbcd24e7a30f66db790ef648f99
>
>
> Cheers!
> --
> Jonathan
>
>
>
> On Sat, Nov 23, 2019 at 8:33 PM Sergi Vladykin <sergi.vladykin@gmail.com>
> wrote:
>
> > Hi!
> >
> > I have two questions related to transactional producers:
> >
> > 1. Is it OK to mix transactional and non-transactional approach with a
> > single KafkaProducer instance? I mean sometimes I want to publish
> multiple
> > messages transactionally, but oftentimes just a single message. Starting
> a
> > transaction for publishing a single message looks inefficient. What is
> the
> > recommend approach here?
> >
> > 2. If I publish multiple messages to multiple partitions in a single
> > transaction is it guaranteed to be all or nothing published? Is it
> possible
> > to end up with only half of the messages published to half of partitions
> in
> > some failure scenario?
> >
> > Sergi
> >
>
>
> --
> Santilli Jonathan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message