kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Adding a new broker behavior
Date Mon, 17 Oct 2011 16:20:22 GMT
Inder,

1. Currently, a topic is created automatically on a broker when it's first
requested (either by a producer or a consumer). Once a topic is created, the
information is registered in ZK.

2. Not sure exactly what you are asking. After each rebalance, each consumer
will get some partitions of a topic to consume from.

3. If the producer uses VIP, it doesn't check ZK. It simply spreads the load
among brokers registered under the vip. This means you can't do
semantic partitioning with the vip approach.


Jun

On Mon, Oct 17, 2011 at 9:01 AM, Inder Pall <inder.pall@gmail.com> wrote:

> Jun,
>
> appreciate your prompt response.
>
> need some more inputs in understanding this better.
> Is the following understanding appropriate.
>
> 1. When a new broker comes in in the cluster, it would register in broker
> registry but won't be subscribed to any topics.
> 2. re-balance triggered by consumers will get some partitions of existing
> topics to this broker.
> 3. If we are using a VIP to front brokers, how do they get all the
> information from ZK about what topics/partitions should go to which broker
> for appropriate routing.
>
> Thanks,
> - Inder
> Yahoo India R&D.
>
> On Mon, Oct 17, 2011 at 8:21 PM, Jun Rao <junrao@gmail.com> wrote:
>
> > Inder,
> >
> > 1. Existing data doesn't get moved to the new broker. Some of the new
> data
> > will get routed to the new broker once the producer recognizes the new
> > broker (either through ZK-based producer or vip).
> >
> > 2. Yes. Once the new broker gets data for a new topic, consumers will be
> > notified and start rebalance.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Oct 16, 2011 at 10:29 PM, Inder Pall <inder.pall@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am trying to understand kafka by going through documents/wiki's for
> > > usage.
> > > Please pardon if the dev mailing list isn't appropriate to be in the TO
> > >  for
> > > this query.
> > >
> > > Couple of queries.
> > >
> > > 1. If you  add new broker boxes, how will existing topics and their
> > > partitions be redistributed on the new broker.
> > > 2. Does [1] trigger a re-balance process via watchers in which case if
> > you
> > > wanna redistribute only certain event types on the new broker, can it
> be
> > > achieved dynamically.
> > >
> > >
> > > -- Inder
> > >
> >
>
>
>
> --
> -- Inder
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message