kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JALADY Christophe <christophe.jal...@laposte.fr>
Subject Kafka Transaction "one by one", what is the bottleneck ?
Date Thu, 12 Mar 2020 09:40:32 GMT
Hello,

I try to understand a performance problem using Transactions:

Here my setup (for whatever reason, we need to create a transaction "per message") (kafka
v2.3/java):

-------------------
consumer.poll().forEach( record -> {
   producer.beginTransaction();
   producer.send(...);
   ...
   producer.sendOffsetsToTransaction(...);
   producer.commitTransaction();
}
-------------------

This setup is _really_ slow (difficult to do better than 20msg/s per partitions). The brokers
seem's not the bottlenek (cpu/io/network thread idle ... everything looks fine enough).

Could someone give me hints:

1) To some magic options to get better performances, client side or broker side if needed
("batching" is not possible right now ... I am aware that it would be though THE solution)

2) To try to understand where the performance is lost ? From my understanding, it may be the
lot of network round trip to achieve the Tx/send/commit/Tx ... but I would like to really
understand what's going on.


Thanks by advance for any answer !


Christophe


Post-scriptum La Poste

Ce message est confidentiel. Sous reserve de tout accord conclu par
ecrit entre vous et La Poste, son contenu ne represente en aucun cas un engagement de la part
de La Poste. Toute publication, utilisation ou diffusion, meme partielle, doit etre autorisee
prealablement. Si vous n'etes pas destinataire de ce message, merci d'en avertir immediatement
l'expediteur.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message