kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Palino <tpal...@linkedin.com>
Subject Re: Kafka Cluster management
Date Wed, 12 Mar 2014 07:10:44 GMT
Both the producer and consumer clients will take care of rebalancing if
there is a broker failure. The consumer handles this entirely in Zookeeper.

For the producer side, you do need some sort of front end to handle the
metadata requests, or else you would constantly have to change your
producer configuration if brokers fail. In our case, we use a load
balancer for this because it's the most robust solution. Only the metadata
request itself will go through the load balancer. Once the producer has
the metadata for the topic and partition, it reconnects directly to the
leader for the partition for producing.

-Todd

On 3/11/14 10:07 PM, "Cassa L" <lcassa8@gmail.com> wrote:

>Hi,
> Is there documentation on how connection management, connection pooling
>works in Kafka for multiple brokers? Do I need to take care of my own
>connection management for cluster? e.g. while publishing messages, if one
>broker stops responding, I need to switch to the other. Does Kafka client
>takes care of this or do I need load balancer between brokers and client?
>
>Thanks,
>LCassa


Mime
View raw message