kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Puneet Lakhina <puneet.lakh...@gmail.com>
Subject Kafka Streams and Scaling Brokers
Date Fri, 19 Jan 2018 20:30:03 GMT

We have a small kafka cluster (3 broker nodes) and as our kafka usage has
grown we are looking to add more brokers. In order for the new brokers to
take on the load of some of the existing topics, I assume we have to either
add partitions that are assigned to these new broker nodes or we have to
move some existing partitions to the new nodes.

We are using kafka streams for some aggregate operations(groupByKey
followed by aggregate) and I wanted to understand how to reason about the
impact of adding/moving partitions on the streams application would be.
Would it be advisable to stop the streams applications, add/move the
nodes/partitions and then resume? Would the preferred method be to move
partitions rather than add in order to maintain the accuracy of the
aggregate operations of kafka streams?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message