kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Santilli <jonathansanti...@gmail.com>
Subject Re: Transactional Producer
Date Sun, 24 Nov 2019 18:11:07 GMT
Hello Sergi,

1. Is it OK to mix transactional and non-transactional approach with a
single KafkaProducer instance?
- This is not possible, a transactional producer can not send data outside
a transaction.

I mean sometimes I want to publish multiple messages transactionally, but
oftentimes just a single message.
Starting a transaction for publishing a single message looks inefficient.
What is the recommend approach here?
- Try to batch the records, if possible, otherwise, you need to begging and
commit the transaction, even for a single record.

2. If I publish multiple messages to multiple partitions in a single
transaction is it guaranteed to be all or nothing published?
- Yes, this is the power of the transactions, all or nothing.

Is it possible to end up with only half of the messages published to half
of partitions in some failure scenario?
- No, this is not possible if you are using correctly a transaction.

Please, take a look at this simple gist with diff scenarios of a
KafkaProducer, hope this help:
https://gist.github.com/jonathansantilli/3b69ebbcd24e7a30f66db790ef648f99


Cheers!
--
Jonathan



On Sat, Nov 23, 2019 at 8:33 PM Sergi Vladykin <sergi.vladykin@gmail.com>
wrote:

> Hi!
>
> I have two questions related to transactional producers:
>
> 1. Is it OK to mix transactional and non-transactional approach with a
> single KafkaProducer instance? I mean sometimes I want to publish multiple
> messages transactionally, but oftentimes just a single message. Starting a
> transaction for publishing a single message looks inefficient. What is the
> recommend approach here?
>
> 2. If I publish multiple messages to multiple partitions in a single
> transaction is it guaranteed to be all or nothing published? Is it possible
> to end up with only half of the messages published to half of partitions in
> some failure scenario?
>
> Sergi
>


-- 
Santilli Jonathan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message