kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastien Falquier <sebastien.falqu...@teads.tv>
Subject How to prevent custom Partitioner from increasing the number of producer's requests?
Date Tue, 02 Jun 2015 07:35:57 GMT
Hi guys,

I am new to Kafka and I am facing a problem I am not able to sort out.

To smooth traffic over all my brokers' partitions, I have coded a custom
Paritioner for my producers, using a simple round robin algorithm that
jumps from a partition to another on every batch of messages (corresponding
to batch.num.messages value). It looks like that :
https://gist.github.com/sfalquier/4c0c7f36dd96d642b416

With that fix, every partitions are used equally, but the amount of
requests from the producers to the brokers have been multiplied by 2. I do
not understand since all producers are async with batch.num.messages=200
and the amount of messages processed is still the same as before. Why do
producers need more requests to do the job? As internal traffic is a bit
critical on our platform, I would really like to reduce producers' requests
volume if possible.

Any idea? Any suggestion?

Regards,
S├ębastien

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message