kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Roesler <j...@confluent.io>
Subject Re: [KafkaStreams 1.1.1] partition assignment broken?
Date Mon, 08 Oct 2018 17:35:40 GMT
Hi Bart,

This sounds a bit surprising. Is there any chance you can zip up some logs
so we can see the assignment protocol on the nodes?

Thanks,
-John

On Mon, Oct 8, 2018 at 4:32 AM Bart Vercammen <bart@cloutrix.com> wrote:

> Hi,
>
> I recently moved some KafkaStreams applications from v0.10.2.1 to v1.1.1
> and now I notice a weird behaviour in the partition assignment.
> When starting 4 instances of my Kafka Streams application (on v1.1.1) I see
> that 17 of the 20 partitions (of a source topic) are assigned to 1 instance
> of the application while the other 3 instances only get 1 partition
> assigned. (previously (on v0.10.2.1) the all got 5 partitions.)
>
> Is this expected behaviour, as I read that quite some improvements were
> done in the partition assignment strategy for Kafka Streams applications?
> If yes, how can I make it so that the partitions are equally devided again
> across all running applications?   It's a bit weird in my opinion as this
> makes scaling the application very hard.
>
> Also, when initially starting with 1 instance of the application, and
> gradually scaling up, the new instances only get 1 partition assigned ...
>
> All my Streams applications use default configuration (more or less),
> running 1 stream-thread.
>
> Any suggestions / enlightenments on this?
> Greets,
> Bart
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message