kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathieu D <matd...@gmail.com>
Subject join not working in relation to how StreamsBuilder builds the topology
Date Mon, 10 Aug 2020 14:44:27 GMT
Dear community,

I have a quite tough problem I struggle to diagnose and fix.
I'm not sure if it's a bug in Kafka-streams or some subtlety I didn't get
in using the DSL api.

The problem is the following.
We have a quite elaborate stream app, working well, in production. We'd
like to add a left join with a KTable (data is coming from a DB via kafka
connect jdbc source).
So we end-up with a topology like this:

Event Source ---- (some transformations and joins) ----- leftJoin(A:
KTable) ----- leftJoin(B: KTable) ---- sinks

The new leftjoin is the one joining A.
The transformations are several custom Transformers.

In tests with TopologyTestDriver, all is good, we can validate the general
logic.
In integration tests with a real Kafka (in a docker, though), we can't
manage to have both left joins work at the same time !
The leftJoin with `A` always return null.

I ran dozen of tests, tweaking and fiddling everything, and I found out
that it's related to partitioning. If I force the number of partitions down
to 1 (by setting all input topics to 1 partition), the join works.
In one of the tests, I suspected one of the transformations, so I removed
it. The topology shown by describe() changed quite significantly (going
from 2 subtoplogies to 1), and this made the leftJoin with A work...and the
leftJoin with B fail.

It drives me crazy.

Activating the optimization didn't help.
The input topics for KTables A and B are read with a TimestamExtractor to
0, since this is static data, to make sure we don't run into timestamp
ordering issues.
We double-checked and tripled-checked the keys in various stages, and we're
sure they're good (by the way, it works with 1 partition).
Partitioner is always the default everywhere (in inputs, kafka-connect...),
actually we never touch this.

Actually it seems related to repartitioning placed in the topology by
StreamsBuilder (probably in relation to transformers ?)

So, I imagine you can't help much without seeing the code, but if you think
of anything that could help diagnosing this further, please tell.

Mathieu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message