kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: partition auto-rebalance
Date Tue, 18 Nov 2014 19:45:20 GMT
The imbalance is measured wrt preferred leaders. i.e., for every
partition, the first replica in the assigned replica list (as shown in
the output of kafka-topics.sh) is called the preferred replica. On
each broker, the auto-balancer counts the number of partitions led by
that broker for which the preferred leader is on another broker. If it
exceeds the thresholds it does a preferred replica leader election.

In your case, can you run the kafka-topics.sh script and see if the
preferred replicas are evenly distributed? Also, which version of
Kafka are you using?

Thanks,

Joel

On Tue, Nov 18, 2014 at 11:48:35AM -0500, Wes Chow wrote:
> I'm trying to understand the config options for auto-rebalancing.
> This is what we have in /etc/kafka/server.properties for all the
> nodes:
> 
> auto.leader.rebalance.enable=true
> leader.imbalance.per.broker.percentage=10
> leader.imbalance.check.interval.seconds=300
> 
> We have 10 nodes for this topic which has 512 partitions. They were
> evenly balanced before I started my experiment. I shut down two of
> the nodes, and the number of leaders per node is now:
> 
>      75 10
>      68 3
>      57 4
>      67 5
>      57 6
>      68 7
>      63 8
>      57 9
> 
> Where the first column is # of leaders, and the second column is
> node #. You can see that nodes 1 and 2 have no leaders, since
> they're down. It's been about half an hour since I did this and the
> balancing hasn't changed.
> 
> The documentation on the config option is very ambiguous. My
> interpretation is that it says if any particular node has 10% more
> leaders then auto-rebalance kicks in. If that means 10% more than
> the average, then node #10 has 75 partitioners, and the average is
> 64, so that's a 17% difference.
> 
> So I think I'm misunderstanding either what auto-rebalance is
> supposed to do or the condition that's supposed to trigger it. Any
> clues?
> 
> Thanks,
> Wes
> 


Mime
View raw message