kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirudh P <panirudh2...@gmail.com>
Subject Question on Using many producers simultaneously
Date Thu, 28 Jan 2016 07:10:04 GMT

We have a scenario where we will be having a large number of producers(
~50k instances - 1 Producer per instance) which will be sending data to a
kafka topic.
The producers could be inactive for a long time and then be asked to send a
message(all at the same time) and as quickly as possible.

What is the best way to do this? Should we keep the Producer connection
open throughout? Or do we create a new Producer each time we need to send a
If the connection is not closed, I believe Kafka discards the connection
after a specified timeout. Also, Does Kafka have any limit on the number of
incoming connection requests? (I read a few blog posts that said that 300k
connections were handled without problems)

Thank you,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message