kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Рябков Алексей Николаевич <a.ryab...@ntc-vulkan.ru>
Subject HA: multi-broker topic rebalance
Date Tue, 27 May 2014 16:04:21 GMT
I found also partition reassignment tool ....but can somebody explain - when I try to execute
reassignment what exactly executed?

I mean when new partition come to new broker what happened  with consumers that connect in
old broker 
1.  they just reconnect to new broker?
2. or they disconnect from partition ?



-----Исходное сообщение-----
От: Рябков Алексей Николаевич 
Отправлено: Tuesday, May 27, 2014 7:38 PM
Кому: users@kafka.apache.org
Тема: HA: multi-broker topic rebalance

I think you don’t understand my problem... I try to explain in more detail....

1. I have a lot of consumers for multiple topics (about 30k)... So one instance of kafka can
not handle it....
2. I try to connect one part of consumers to one instance of kafka (server 1) and another
part  to other (server 2) and rest of consumers to server 3 3. When producers try to add message
to every topic I use round-robin (so first message add to server 1, second to server 2 etc...)
4. After some period all consumers (from server 1) report that   they have no message in topic....
But other consumers (from server 2 and 3) have a lot of message in this topic...
   
So how I can rebalance topic between servers?
I can not just switch (for example )all consumers to server 2 ...because every server can
not handle 20 k clients :)  I must create more clever solution...
What can I do?...
1.  If producer can read statistics how many unread message in server exists I can put message
to most free server (not round-robin)  But I can not find any interface for that...may be
I missed it?
2. May be exist something like federated plugin for rabbitmq - http://www.rabbitmq.com/federation.html
 (or how can I create something similar.... what kind of API can I use?)

Thanks in advance, Aleksey Ryabkov

-----Исходное сообщение-----
От: Jun Rao [mailto:junrao@gmail.com]
Отправлено: Tuesday, May 27, 2014 6:26 PM
Кому: users@kafka.apache.org
Тема: Re: multi-broker topic rebalance

I suspect the issue is that since you have many more partitions than consumers, the data distribution
among the partitions is not even. You can take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

Thanks,

Jun


On Mon, May 26, 2014 at 8:45 PM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru>
wrote:

> Hello!
> >>Which version of Kafka are you using?
>  1. I am using 0.8.1
> >>How many partitions do you have in that topic?
> 2. In docs I found that number of partition must be equal or greater 
> than number of consumers ... so I try 2 case:
>     - 10k partition per broker
>     - 20k partition per broker
>
> And one more question - how to  force rebalance partitions to 
> consumers in borker (and how much it cost...) I found in doc that 
> consumer can "Force itself to rebalance within in its consumer group"
> but can not find API call...So for now I just reconnect....
>
> With best regards, Aleksey Ryabkov
>
> -----Исходное сообщение-----
> От: Jun Rao [mailto:junrao@gmail.com]
> Отправлено: Tuesday, May 27, 2014 7:29 AM
> Кому: users@kafka.apache.org
> Тема: Re: multi-broker topic rebalance
>
> Which version of Kafka are you using? How many partitions do you have 
> in that topic?
>
> Thanks,
>
> Jun
>
>
> On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич <

> a.ryabkov@ntc-vulkan.ru> wrote:
>
> >
> > Hello!
> >
> > How can I rebalance topic within multiple brokers?
> > For example:
> >
> > 1.        I have about 30k consumers...
> >
> > 2.        I need to use 3 broker (10k consumer per broker)
> >
> > 3.       I use multiple (around 10) producers with default balance
> strategy
> > Sometimes one broker have empty topic (consumer  finished all 
> > message on broker)...But another broker have a lot of messages in topic...
> >
> > How to resolve such situation:
> > -i cant just switch my consumers to another broker ( broker can not 
> > handle more than 10k consumers....) -I can not find how to rebalance 
> > topic within multiple brokers...
> > - I can not find how to setup limit for message per topic (or
> > partition) ... If such things exists I can handle such case when add 
> > new message to broker in consumer side...and switch to  another ...
> >
> > Thanks in advance, Aleksey Ryabkov
> >
>
Mime
View raw message