kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Jablonski <ajablon...@thoughtworks.com>
Subject Kafka Streams application stuck rebalancing on startup
Date Fri, 16 Oct 2020 05:00:57 GMT
Hey there!

My team and I have run across a bit of a jam in our application where,
given a particular setup, our Kafka Streams application never seems to
start successfully, instead just getting stuck in the REBALANCING state.
We've been able to get the crucial factors that cause this behavior down to
a particular combination of (1) grouping, (2) windowing, (3) aggregating,
and (4) foreign-key joining, with some of those steps specifying Serdes
besides the default.

It's probably more useful to see a minimal example, so there's one here
<https://github.com/ajablonski/streams-issue-demo/blob/master/build.gradle>.
The underlying Kafka Streams version is 2.5.1. The first test should show
the application eventually transition to running state, but it doesn't
within the 30 second timeout I've set. Interestingly, getting rid of the
'Grouped.with' argument to the 'groupBy' function and the
'Materialized.with' in 'aggregate' in the 'StreamsConfiguration' lets the
application transition to "RUNNING", though without the correct Serdes
that's not too valuable.

There might be a cleaner way to organize the particular flow in the toy
example, but is there something fundamentally wrong with the approach laid
out in that application that would cause Streams to be stuck in
REBALANCING? I'd appreciate any advice folks could give!

Thanks!
Alex Jablonski

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message