kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kafka user <kafkaraic...@gmail.com>
Subject Re: kafka cluster behind a hardware load balancer
Date Tue, 01 Apr 2014 05:16:25 GMT
Thanks Jun for your quick reply.

If I understood correctly,  for the first step ( fetching metadata ),
producer can still go through a VIP whereas for the second step ( once
producer gets the metadata ),  producer needs to connect to the correct
instance of the broker that hosts the partition.   I guess for the second
step, producer should be able to directly connect to the right broker and
this cant be achieved through a Load Balancer ( as LB may send the request
to any one of the machines in the pool ).

Is my understanding correct ? Does it also mean that when a new broker is
added or an existing broker is removed, we need to keep the list of brokers
updated ?


On Mon, Mar 31, 2014 at 9:22 PM, Jun Rao <junrao@gmail.com> wrote:

> In Kafka, the broker list you provide to the producer is used only for
> fetching metadata, which can be served on any broker. The producer then
> connects to the right brokers according to the metadata response.
> Thanks,
> Jun
> On Mon, Mar 31, 2014 at 6:04 PM, kafka user <kafkaraichur@gmail.com>
> wrote:
> > If we have a kafka producer connecting to a Kafka cluster behind a
> hardware
> > Load Balancer (VIP), will producer be able to send a message to a right
> > partition  ?
> > Can one of the brokers in a cluster do a broker discovery to forward the
> > message to ?
> > I guess my question is whether it makes sense to use a hardware load
> > balancer at all.
> >
> > Typical deployment model at our company is to have a pool of machines and
> > LB decides which machine in a pool would receive a request. Looking to
> hear
> > from kafka experts what is the best way of setting up a cluster given
> that
> > the cluster has be behind a VIP.
> >
> >
> > Thanks
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message