kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Рябков Алексей Николаевич <a.ryab...@ntc-vulkan.ru>
Subject HA: multi-broker topic rebalance
Date Tue, 27 May 2014 03:45:25 GMT
>>Which version of Kafka are you using?
 1. I am using 0.8.1
>>How many partitions do you have in that topic?
2. In docs I found that number of partition must be equal or greater than number of consumers
... so I try 2 case:
    - 10k partition per broker
    - 20k partition per broker

And one more question - how to  force rebalance partitions to consumers in borker (and how
much it cost...) 
I found in doc that consumer can "Force itself to rebalance within in its consumer group"
but can not find API call...So for now I just reconnect....

With best regards, Aleksey Ryabkov

-----Исходное сообщение-----
От: Jun Rao [mailto:junrao@gmail.com] 
Отправлено: Tuesday, May 27, 2014 7:29 AM
Кому: users@kafka.apache.org
Тема: Re: multi-broker topic rebalance

Which version of Kafka are you using? How many partitions do you have in that topic?



On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru>

> Hello!
> How can I rebalance topic within multiple brokers?
> For example:
> 1.        I have about 30k consumers...
> 2.        I need to use 3 broker (10k consumer per broker)
> 3.       I use multiple (around 10) producers with default balance strategy
> Sometimes one broker have empty topic (consumer  finished all message 
> on broker)...But another broker have a lot of messages in topic...
> How to resolve such situation:
> -i cant just switch my consumers to another broker ( broker can not 
> handle more than 10k consumers....) -I can not find how to rebalance 
> topic within multiple brokers...
> - I can not find how to setup limit for message per topic (or 
> partition) ... If such things exists I can handle such case when add 
> new message to broker in consumer side...and switch to  another ...
> Thanks in advance, Aleksey Ryabkov
View raw message