kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cassa L <lcas...@gmail.com>
Subject Re: Kafka Cluster management
Date Mon, 17 Mar 2014 17:45:01 GMT
Thanks. I checked in my sample code that requests have been distributed to
brokers evenly without load balancers.

..Leena



On Wed, Mar 12, 2014 at 3:34 PM, Todd Palino <tpalino@linkedin.com> wrote:

> I apologize, you are correct. I forgot that you can pass in a list of
> hosts. Using a balancer is the way we handle it here, mostly because we
> have many producers, and they are not all under our direct control.
>
> To answer the question of what load balancer, we have hardware load
> balancers in place in our environment. I'm not certain of the model, as I
> generally don't interface with them directly (that's the purview of the
> network operations team).
>
> -Todd
>
> On 3/12/14 1:47 PM, "Cassa L" <lcassa8@gmail.com> wrote:
>
> >Hi Todd,
> >  I am not very clear about you mentioned - "Both the producer and
> >consumer
> >clients will take care of rebalancing if
> >there is a broker failure." "
> >For the producer side, you do need some sort of front end to handle the
> >metadata requests, or else you would constantly have to change your
> >producer configuration if brokers fail."
> >
> > Isn't broker client API needs to take care of collecting metadata based
> >on
> >the list of brokers that I pass in producer API? Why would I need load
> >balancer just to get metadata?
> >
> >Regards,
> >LCass
> >
> >
> >On Wed, Mar 12, 2014 at 2:39 AM, Atig <andthereitgoes@hotmail.com> wrote:
> >
> >> Hello
> >>
> >> Can you elaborate on what is your producer setup in terms of fetching
> >>the
> >> metadata (in conjunction with a load balancer) and some sort of code
> >> skeleton? New to Kafka here.
> >>
> >> Also, if you don't mind sharing what do you use as load balancer in this
> >> scenario?
> >>
> >> Thanks
> >>
> >> --- Original Message ---
> >>
> >> From: "Todd Palino" <tpalino@linkedin.com>
> >> Sent: 12 March 2014 07:11
> >> To: users@kafka.apache.org
> >> Subject: Re: Kafka Cluster management
> >>
> >> Both the producer and consumer clients will take care of rebalancing if
> >> there is a broker failure. The consumer handles this entirely in
> >>Zookeeper.
> >>
> >> For the producer side, you do need some sort of front end to handle the
> >> metadata requests, or else you would constantly have to change your
> >> producer configuration if brokers fail. In our case, we use a load
> >> balancer for this because it's the most robust solution. Only the
> >>metadata
> >> request itself will go through the load balancer. Once the producer has
> >> the metadata for the topic and partition, it reconnects directly to the
> >> leader for the partition for producing.
> >>
> >> -Todd
> >>
> >> On 3/11/14 10:07 PM, "Cassa L" <lcassa8@gmail.com> wrote:
> >>
> >> >Hi,
> >> > Is there documentation on how connection management, connection
> >>pooling
> >> >works in Kafka for multiple brokers? Do I need to take care of my own
> >> >connection management for cluster? e.g. while publishing messages, if
> >>one
> >> >broker stops responding, I need to switch to the other. Does Kafka
> >>client
> >> >takes care of this or do I need load balancer between brokers and
> >>client?
> >> >
> >> >Thanks,
> >> >LCassa
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message