kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From h...@confluent.io
Subject Re: Kafka balancing
Date Fri, 06 Jan 2017 18:55:18 GMT
You can use the partition reassignment tool to move larger partitions from the full node over
to the lighter used nodes.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-Selectivelymovingsomepartitionstoabroker

There are also some open source and commercial tools that will rebalance the entire cluster
for you automatically and with less time and effort than manually moving around individual
partitions. 

https://www.confluent.io/product/auto-data-balancing/

If you are running a release older than 0.10 then you should be careful to perform any rebalancing
in off hours as the replication traffic can negatively impact the production traffic. In the
latest releases there is a feature to separately throttle the replication traffic separately
from client traffic.

-hans

> On Jan 6, 2017, at 12:23 PM, R . <trisonicking@msn.com> wrote:
> 
> Hello, I have a 3node cluster on identical server setups.  We've noticed that one of
the kafka's write a lot more data than the other two.  Recently, that kafka server has completely
filled our data partition while the other two kafka servers were still at, for example, 30%
capacity.  Is there a config that needs tuning? How do I balance out the servers so that they
all more or less have a similar capacity of storage after processing messages? Thank you in
advance.

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message