kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: multi-broker topic rebalance
Date Tue, 27 May 2014 14:26:28 GMT
I suspect the issue is that since you have many more partitions than
consumers, the data distribution among the partitions is not even. You can
take a look at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

Thanks,

Jun


On Mon, May 26, 2014 at 8:45 PM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:

> Hello!
> >>Which version of Kafka are you using?
>  1. I am using 0.8.1
> >>How many partitions do you have in that topic?
> 2. In docs I found that number of partition must be equal or greater than
> number of consumers ... so I try 2 case:
>     - 10k partition per broker
>     - 20k partition per broker
>
> And one more question - how to  force rebalance partitions to consumers in
> borker (and how much it cost...)
> I found in doc that consumer can "Force itself to rebalance within in its
> consumer group" but can not find API call...So for now I just reconnect....
>
> With best regards, Aleksey Ryabkov
>
> -----Исходное сообщение-----
> От: Jun Rao [mailto:junrao@gmail.com]
> Отправлено: Tuesday, May 27, 2014 7:29 AM
> Кому: users@kafka.apache.org
> Тема: Re: multi-broker topic rebalance
>
> Which version of Kafka are you using? How many partitions do you have in
> that topic?
>
> Thanks,
>
> Jun
>
>
> On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> >
> > Hello!
> >
> > How can I rebalance topic within multiple brokers?
> > For example:
> >
> > 1.        I have about 30k consumers...
> >
> > 2.        I need to use 3 broker (10k consumer per broker)
> >
> > 3.       I use multiple (around 10) producers with default balance
> strategy
> > Sometimes one broker have empty topic (consumer  finished all message
> > on broker)...But another broker have a lot of messages in topic...
> >
> > How to resolve such situation:
> > -i cant just switch my consumers to another broker ( broker can not
> > handle more than 10k consumers....) -I can not find how to rebalance
> > topic within multiple brokers...
> > - I can not find how to setup limit for message per topic (or
> > partition) ... If such things exists I can handle such case when add
> > new message to broker in consumer side...and switch to  another ...
> >
> > Thanks in advance, Aleksey Ryabkov
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message