kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Duggan <bryan.dug...@boxever.com>
Subject Problems trying to make kafka 'rack-aware'
Date Thu, 20 Sep 2018 10:31:53 GMT

I have a kafka cluster consisting of 3 brokers across 3 different AWS 
availability zones.  It hosts several topics, each of which has a 
replication factor of 3. The cluster is currently not 'rack-aware'.

I am trying to do the following;

     - add 3 additional brokers (one in each of the 3 AZs)

     - make the cluster 'rack-aware'. (ie: create 3 racks on a per-AZ 
basic, each containing 2 brokers)

     - reassign the topics with the intention of having 1 replica in 
each of the 3 racks.

To achieve this I've added 'broker.rack' to the properties file for each 
broker. The rack name is the same as the AZ name each broker is in. I've 
restarted kafka on all brokers (in case that's required for 
rack-awareness to take effect).

Following restart I've attempted to reassign topics across all 6 brokers 
by running the following;

     - ./kafka-reassign-partitions.sh --zookeeper $ZK 
--topics-to-move-json-file topics-to-move.json --broker-list '1,2,3,4,5,6'

(where topics-to-move.json is a simple json file containing the topics 
to reassign)

The problem I am having is, after running 'kafka-reassign-partitions.sh' 
with 6 brokers listed in the broker-list, it doesn't honour  
rack-awareness, and instead assigns 2 partitions to brokers in a single 
rack with a 3rd being assigned elsewhere.

The version of kafka I am using is 2.11-1.1.1.

Any documentation I've read suggests the above should have achieved what 
I want. However, it is not working as expected.

Has anyone else make their kafka cluster 'rack-aware'? If so, did you 
experience any issues doing so?

Or, can anyone tell me if there's some step I'm missing to make this work.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message