kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: multi-broker topic rebalance
Date Tue, 27 May 2014 16:44:37 GMT
You can read about the partition assignment tool
here<http://kafka.apache.org/081/documentation.html#basic_ops_cluster_expansion>.
It moves data for a partition's replica from one broker to the other.

Thanks,
Neha



On Tue, May 27, 2014 at 9:04 AM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:

> I found also partition reassignment tool ....but can somebody explain -
> when I try to execute reassignment what exactly executed?
>
> I mean when new partition come to new broker what happened  with consumers
> that connect in old broker
> 1.  they just reconnect to new broker?
> 2. or they disconnect from partition ?
>
>
>
> -----Исходное сообщение-----
> От: Рябков Алексей Николаевич
> Отправлено: Tuesday, May 27, 2014 7:38 PM
> Кому: users@kafka.apache.org
> Тема: HA: multi-broker topic rebalance
>
> I think you don’t understand my problem... I try to explain in more
> detail....
>
> 1. I have a lot of consumers for multiple topics (about 30k)... So one
> instance of kafka can not handle it....
> 2. I try to connect one part of consumers to one instance of kafka (server
> 1) and another part  to other (server 2) and rest of consumers to server 3
> 3. When producers try to add message to every topic I use round-robin (so
> first message add to server 1, second to server 2 etc...)
> 4. After some period all consumers (from server 1) report that   they have
> no message in topic.... But other consumers (from server 2 and 3) have a
> lot of message in this topic...
>
> So how I can rebalance topic between servers?
> I can not just switch (for example )all consumers to server 2 ...because
> every server can not handle 20 k clients :)  I must create more clever
> solution...
> What can I do?...
> 1.  If producer can read statistics how many unread message in server
> exists I can put message to most free server (not round-robin)  But I can
> not find any interface for that...may be I missed it?
> 2. May be exist something like federated plugin for rabbitmq -
> http://www.rabbitmq.com/federation.html  (or how can I create something
> similar.... what kind of API can I use?)
>
> Thanks in advance, Aleksey Ryabkov
>
> -----Исходное сообщение-----
> От: Jun Rao [mailto:junrao@gmail.com]
> Отправлено: Tuesday, May 27, 2014 6:26 PM
> Кому: users@kafka.apache.org
> Тема: Re: multi-broker topic rebalance
>
> I suspect the issue is that since you have many more partitions than
> consumers, the data distribution among the partitions is not even. You can
> take a look at
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
> ?
>
> Thanks,
>
> Jun
>
>
> On Mon, May 26, 2014 at 8:45 PM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > Hello!
> > >>Which version of Kafka are you using?
> >  1. I am using 0.8.1
> > >>How many partitions do you have in that topic?
> > 2. In docs I found that number of partition must be equal or greater
> > than number of consumers ... so I try 2 case:
> >     - 10k partition per broker
> >     - 20k partition per broker
> >
> > And one more question - how to  force rebalance partitions to
> > consumers in borker (and how much it cost...) I found in doc that
> > consumer can "Force itself to rebalance within in its consumer group"
> > but can not find API call...So for now I just reconnect....
> >
> > With best regards, Aleksey Ryabkov
> >
> > -----Исходное сообщение-----
> > От: Jun Rao [mailto:junrao@gmail.com]
> > Отправлено: Tuesday, May 27, 2014 7:29 AM
> > Кому: users@kafka.apache.org
> > Тема: Re: multi-broker topic rebalance
> >
> > Which version of Kafka are you using? How many partitions do you have
> > in that topic?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич
<
> > a.ryabkov@ntc-vulkan.ru> wrote:
> >
> > >
> > > Hello!
> > >
> > > How can I rebalance topic within multiple brokers?
> > > For example:
> > >
> > > 1.        I have about 30k consumers...
> > >
> > > 2.        I need to use 3 broker (10k consumer per broker)
> > >
> > > 3.       I use multiple (around 10) producers with default balance
> > strategy
> > > Sometimes one broker have empty topic (consumer  finished all
> > > message on broker)...But another broker have a lot of messages in
> topic...
> > >
> > > How to resolve such situation:
> > > -i cant just switch my consumers to another broker ( broker can not
> > > handle more than 10k consumers....) -I can not find how to rebalance
> > > topic within multiple brokers...
> > > - I can not find how to setup limit for message per topic (or
> > > partition) ... If such things exists I can handle such case when add
> > > new message to broker in consumer side...and switch to  another ...
> > >
> > > Thanks in advance, Aleksey Ryabkov
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message