kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dana Powers <dana.pow...@rd.io>
Subject Re: Load Balancing Kafka
Date Thu, 16 Jul 2015 14:57:03 GMT
I think the answer here is that the Kafka protocol includes a broker
metadata api. The client uses the broker host(s) you provide to discover
the full list of brokers in the cluster (and the topics+partitions each
manages/leads). The java client has a similar interface via
metadata.brokers.list / bootstrap.servers.

-Dana
AhŠ It seems you are more focusing on producer side workload balanceŠ If
that is the case, please ignore my previous comments.

Jiangjie (Becket) Qin

On 7/15/15, 6:01 PM, "Jiangjie Qin" <jqin@linkedin.com> wrote:

>If you have pretty balanced traffic on each partition and have set
>auto.leader.rebalance.enabled to true or false, you might not need to do
>further workload balance.
>
>However, in most cases you probably still need to do some sort of load
>balancing based on the traffic and disk utilization of each broker. You
>might want to do leader migration and/or partition reassignment.
>
>Leader migration is a cheaper rebalance and mostly addresses CPU and
>Network unbalance. Partition reassignment is a much more expensive
>operation as it moves actual data, this can help with disk utilization in
>addition to CPU and network.
>
>Thanks,
>
>Jiangjie (Becket) Qin
>
>On 7/15/15, 5:19 PM, "Sandy Waters" <sandy.watermellons@gmail.com> wrote:
>
>>Hi all,
>>
>>Do I need to load balance against the brokers?  I am using the python
>>driver and it seems to only want a single kafka broker host.  However, in
>>a
>>situation where I have 10 brokers, is it still fine to just give it one
>>host.  Does zookeeper and kafka handle the load balancing and redirect my
>>push somewhere else?
>>
>>Would it hurt if I load balanced with Nginx and had it do round robin to
>>the brokers?
>>
>>Much thanks for any help.
>>
>>-Sandy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message