kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bart Vercammen <b...@cloutrix.com>
Subject [KafkaStreams 1.1.1] partition assignment broken?
Date Mon, 08 Oct 2018 09:32:27 GMT
Hi,

I recently moved some KafkaStreams applications from v0.10.2.1 to v1.1.1
and now I notice a weird behaviour in the partition assignment.
When starting 4 instances of my Kafka Streams application (on v1.1.1) I see
that 17 of the 20 partitions (of a source topic) are assigned to 1 instance
of the application while the other 3 instances only get 1 partition
assigned. (previously (on v0.10.2.1) the all got 5 partitions.)

Is this expected behaviour, as I read that quite some improvements were
done in the partition assignment strategy for Kafka Streams applications?
If yes, how can I make it so that the partitions are equally devided again
across all running applications?   It's a bit weird in my opinion as this
makes scaling the application very hard.

Also, when initially starting with 1 instance of the application, and
gradually scaling up, the new instances only get 1 partition assigned ...

All my Streams applications use default configuration (more or less),
running 1 stream-thread.

Any suggestions / enlightenments on this?
Greets,
Bart

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message