kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ross <dyr...@klout.com>
Subject How fair should I expect rebalancing to be?
Date Thu, 13 Dec 2012 07:17:55 GMT
Hello,

I am trying to distribute work across several nodes using Kafka. I have 3
brokers each with 16 partitions. I have 8 worker servers listening with one
message stream on the same topic. I expect each server to own about 1/8 of
the partitions, yet I am not seeing this. It seems initially, the work is
fairly evenly distributed. However, after running for several hours, I see
that only three consumers own any partitions, and only 32 of the 48 have an
owner at all.

What gives?

For reference, we have 0.7.0 on the server and 0.7.2 on the consumer. Also,
I set the max rebalance retries to be 10 because I saw a lot of rebalance
failures in the logs.


Thanks,

David

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message