kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: How fair should I expect rebalancing to be?
Date Thu, 13 Dec 2012 16:03:07 GMT
Do you see "fail to rebalance after 4 tries" in the worker not getting any
data? If so, what's the ZK version? You should use 3.3.4 or above, which
fixed some bugs that could cause rebalance to fail.

Thanks,

Jun

On Wed, Dec 12, 2012 at 11:17 PM, David Ross <dyross@klout.com> wrote:

> Hello,
>
> I am trying to distribute work across several nodes using Kafka. I have 3
> brokers each with 16 partitions. I have 8 worker servers listening with one
> message stream on the same topic. I expect each server to own about 1/8 of
> the partitions, yet I am not seeing this. It seems initially, the work is
> fairly evenly distributed. However, after running for several hours, I see
> that only three consumers own any partitions, and only 32 of the 48 have an
> owner at all.
>
> What gives?
>
> For reference, we have 0.7.0 on the server and 0.7.2 on the consumer. Also,
> I set the max rebalance retries to be 10 because I saw a lot of rebalance
> failures in the logs.
>
>
> Thanks,
>
> David
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message