kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ara Ebrahimi <ara.ebrah...@argyledata.com>
Subject kafka streams consumer partition assignment is uneven
Date Mon, 09 Jan 2017 17:52:52 GMT
Hi,

I have 3 kafka brokers, each with 4 disks. I have 12 partitions. I have 3 kafka streams nodes.
Each is configured to have 4 streaming threads. My topology is quite complex and I have 7
topics and lots of joins and states.

What I have noticed is that each of the 3 kafka streams nodes gets configured to process variables
number of partitions of a topic. One node is assigned to process 2 partitions of topic a and
another one gets assigned 5. Hence I end up with nonuniform throughput across these nodes.
One node ends up processing more data than the other.

What’s going on? How can I make sure partitions assignment to kafka streams nodes is uniform?

On a similar topic, is there a way to make sure partition assignment to disks across kafka
brokers is also uniform? Even if I use a round-robin one to pin partitions to broker, but
there doesn’t seem to be a way to uniformly pin partitions to disks. Or maybe I’m missing
something here? I end up with 2 partitions of topic a on disk 1 and 3 partitions of topic
a on disk 2. It’s a bit variable. Not totally random, but it’s not uniformly distributed
either.

Ara.



________________________________

This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise confidential information. If you have received it in error, please notify the
sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
Thank you in advance for your cooperation.

________________________________
Mime
View raw message