kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bart Vercammen <b...@cloutrix.com>
Subject Re: [KafkaStreams 1.1.1] partition assignment broken?
Date Mon, 08 Oct 2018 18:23:49 GMT
Hi John,

Zipping up some logs from our running Kafka cluster is going to be a bit
difficult.
What I can do is try to reproduce this off-line and capture the logs from
there.

We also had a look in the PartitionAssignor source code (for 1.1.1) and
indeed this behaviour is a bit weird
as from the source code I'd expect equally divided partitions.

Anyway, hopefully I'll be able to reproduce this issue with some simple
unit-test like code.
I'll post the results when I have more info.

Greets,
Bart

On Mon, Oct 8, 2018 at 7:36 PM John Roesler <john@confluent.io> wrote:

> Hi Bart,
>
> This sounds a bit surprising. Is there any chance you can zip up some logs
> so we can see the assignment protocol on the nodes?
>
> Thanks,
> -John
>
> On Mon, Oct 8, 2018 at 4:32 AM Bart Vercammen <bart@cloutrix.com> wrote:
>
> > Hi,
> >
> > I recently moved some KafkaStreams applications from v0.10.2.1 to v1.1.1
> > and now I notice a weird behaviour in the partition assignment.
> > When starting 4 instances of my Kafka Streams application (on v1.1.1) I
> see
> > that 17 of the 20 partitions (of a source topic) are assigned to 1
> instance
> > of the application while the other 3 instances only get 1 partition
> > assigned. (previously (on v0.10.2.1) the all got 5 partitions.)
> >
> > Is this expected behaviour, as I read that quite some improvements were
> > done in the partition assignment strategy for Kafka Streams applications?
> > If yes, how can I make it so that the partitions are equally devided
> again
> > across all running applications?   It's a bit weird in my opinion as this
> > makes scaling the application very hard.
> >
> > Also, when initially starting with 1 instance of the application, and
> > gradually scaling up, the new instances only get 1 partition assigned ...
> >
> > All my Streams applications use default configuration (more or less),
> > running 1 stream-thread.
> >
> > Any suggestions / enlightenments on this?
> > Greets,
> > Bart
> >
>


-- 
Mvg,
Bart Vercammen


clouTrix BVBA
+32 486 69 17 68
info@cloutrix.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message