kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: kafka cluster behind a hardware load balancer
Date Tue, 01 Apr 2014 15:02:02 GMT
The second step can't be done through a LB. The produce will have to
connect to the right broker directly. We don't do request forwarding in the
brokers.

Also, I added
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowshouldIsetmetadata.broker.list?
See if that answers your other question.

Thanks,

Jun


On Mon, Mar 31, 2014 at 10:16 PM, kafka user <kafkaraichur@gmail.com> wrote:

> Thanks Jun for your quick reply.
>
> If I understood correctly,  for the first step ( fetching metadata ),
> producer can still go through a VIP whereas for the second step ( once
> producer gets the metadata ),  producer needs to connect to the correct
> instance of the broker that hosts the partition.   I guess for the second
> step, producer should be able to directly connect to the right broker and
> this cant be achieved through a Load Balancer ( as LB may send the request
> to any one of the machines in the pool ).
>
> Is my understanding correct ? Does it also mean that when a new broker is
> added or an existing broker is removed, we need to keep the list of brokers
> updated ?
>
> Thanks
>
>
>
>
>
>
>
>
>
>
> On Mon, Mar 31, 2014 at 9:22 PM, Jun Rao <junrao@gmail.com> wrote:
>
> > In Kafka, the broker list you provide to the producer is used only for
> > fetching metadata, which can be served on any broker. The producer then
> > connects to the right brokers according to the metadata response.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Mar 31, 2014 at 6:04 PM, kafka user <kafkaraichur@gmail.com>
> > wrote:
> >
> > > If we have a kafka producer connecting to a Kafka cluster behind a
> > hardware
> > > Load Balancer (VIP), will producer be able to send a message to a right
> > > partition  ?
> > > Can one of the brokers in a cluster do a broker discovery to forward
> the
> > > message to ?
> > > I guess my question is whether it makes sense to use a hardware load
> > > balancer at all.
> > >
> > > Typical deployment model at our company is to have a pool of machines
> and
> > > LB decides which machine in a pool would receive a request. Looking to
> > hear
> > > from kafka experts what is the best way of setting up a cluster given
> > that
> > > the cluster has be behind a VIP.
> > >
> > >
> > > Thanks
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message