kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Kumar <varunkumar...@hotmail.com>
Subject Scaling Kafka Cluster in Cloud
Date Mon, 03 Jun 2019 03:48:10 GMT
Hi,

I have a Kafka cluster setup in cloud, I tried a different approach to scale it.

I gracefully shutdown a broker with 2 (or more) disks and attach 2 brokers with 1 (or equally
distributed) disks each. (Had to use partition reassignment tool change partitions after new
broker is attached).

It actually worked with Kafka 0.11 where after reassigning partition, only the new data was
copied to broker and broker added to ISR much faster than the approach given in documentation<https://kafka.apache.org/documentation/#basic_ops_cluster_expansion>.
I understand for HA I need at least 3 replicas and all partitions in broker which will be
scaled needs to have atleast 2 ISR other than the scaling broker.

Would like to know if this approach would work in production ? Any scenarios where data might
be to data loss ? And also if it will actually work in latest version of Kafka ?

Thanks
Varun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message