kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cassa L <lcas...@gmail.com>
Subject Re: Kafka Cluster management
Date Wed, 12 Mar 2014 20:47:40 GMT
Hi Todd,
  I am not very clear about you mentioned - "Both the producer and consumer
clients will take care of rebalancing if
there is a broker failure." "
For the producer side, you do need some sort of front end to handle the
metadata requests, or else you would constantly have to change your
producer configuration if brokers fail."

 Isn't broker client API needs to take care of collecting metadata based on
the list of brokers that I pass in producer API? Why would I need load
balancer just to get metadata?

Regards,
LCass


On Wed, Mar 12, 2014 at 2:39 AM, Atig <andthereitgoes@hotmail.com> wrote:

> Hello
>
> Can you elaborate on what is your producer setup in terms of fetching the
> metadata (in conjunction with a load balancer) and some sort of code
> skeleton? New to Kafka here.
>
> Also, if you don't mind sharing what do you use as load balancer in this
> scenario?
>
> Thanks
>
> --- Original Message ---
>
> From: "Todd Palino" <tpalino@linkedin.com>
> Sent: 12 March 2014 07:11
> To: users@kafka.apache.org
> Subject: Re: Kafka Cluster management
>
> Both the producer and consumer clients will take care of rebalancing if
> there is a broker failure. The consumer handles this entirely in Zookeeper.
>
> For the producer side, you do need some sort of front end to handle the
> metadata requests, or else you would constantly have to change your
> producer configuration if brokers fail. In our case, we use a load
> balancer for this because it's the most robust solution. Only the metadata
> request itself will go through the load balancer. Once the producer has
> the metadata for the topic and partition, it reconnects directly to the
> leader for the partition for producing.
>
> -Todd
>
> On 3/11/14 10:07 PM, "Cassa L" <lcassa8@gmail.com> wrote:
>
> >Hi,
> > Is there documentation on how connection management, connection pooling
> >works in Kafka for multiple brokers? Do I need to take care of my own
> >connection management for cluster? e.g. while publishing messages, if one
> >broker stops responding, I need to switch to the other. Does Kafka client
> >takes care of this or do I need load balancer between brokers and client?
> >
> >Thanks,
> >LCassa
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message