kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: Adding a new broker behavior
Date Mon, 17 Oct 2011 16:32:06 GMT
thanks guys, it's clear now.

- inder



On Mon, Oct 17, 2011 at 9:51 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Inder,
>
> From the producer side,VIP and the ZK Producer are mutually exclusive
> options.
> You use either one or the other.
>
> > 1. When a new broker comes in in the cluster, it would register in broker
> > registry but won't be subscribed to any topics.
>
> If you are using a VIP, then it will route some of the new requests to the
> new
> broker automatically. If you are using the ZK Producer, it will get
> notified once the
> new broker registers itself in ZK, and it will start routing new
> requests to the broker
> according to the partitioner you use (Default is random partitioning,
> which is similar to
> VIP behavior)
>
> > 3. If we are using a VIP to front brokers, how do they get all the
> > information from ZK about what topics/partitions should go to which
> broker
> > for appropriate routing.
>
> As I said, if you are using VIP, then brokers are not "discovered"
> through ZK watchers.
> The VIP should have a TCP healthcheck on the Kafka port, and all
> brokers that have
> Kafka running on that port, will be involved in receiving Kafka data.
> The VIP randomly
> distributes produce requests amongst the live brokers.
>
> Thanks,
> Neha
>
>
> On Mon, Oct 17, 2011 at 9:01 AM, Inder Pall <inder.pall@gmail.com> wrote:
> > Jun,
> >
> > appreciate your prompt response.
> >
> > need some more inputs in understanding this better.
> > Is the following understanding appropriate.
> >
> > 1. When a new broker comes in in the cluster, it would register in broker
> > registry but won't be subscribed to any topics.
> > 2. re-balance triggered by consumers will get some partitions of existing
> > topics to this broker.
> > 3. If we are using a VIP to front brokers, how do they get all the
> > information from ZK about what topics/partitions should go to which
> broker
> > for appropriate routing.
> >
> > Thanks,
> > - Inder
> > Yahoo India R&D.
> >
> > On Mon, Oct 17, 2011 at 8:21 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> Inder,
> >>
> >> 1. Existing data doesn't get moved to the new broker. Some of the new
> data
> >> will get routed to the new broker once the producer recognizes the new
> >> broker (either through ZK-based producer or vip).
> >>
> >> 2. Yes. Once the new broker gets data for a new topic, consumers will be
> >> notified and start rebalance.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Sun, Oct 16, 2011 at 10:29 PM, Inder Pall <inder.pall@gmail.com>
> wrote:
> >>
> >> > Hi,
> >> >
> >> > I am trying to understand kafka by going through documents/wiki's for
> >> > usage.
> >> > Please pardon if the dev mailing list isn't appropriate to be in the
> TO
> >> >  for
> >> > this query.
> >> >
> >> > Couple of queries.
> >> >
> >> > 1. If you  add new broker boxes, how will existing topics and their
> >> > partitions be redistributed on the new broker.
> >> > 2. Does [1] trigger a re-balance process via watchers in which case if
> >> you
> >> > wanna redistribute only certain event types on the new broker, can it
> be
> >> > achieved dynamically.
> >> >
> >> >
> >> > -- Inder
> >> >
> >>
> >
> >
> >
> > --
> > -- Inder
> >
>



-- 
-- Inder

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message