kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edmondo Porcu <edmondo.po...@gmail.com>
Subject How does KStream transform performs repartitioning?
Date Tue, 22 May 2018 10:12:46 GMT
Hello users,

we are performing a Transform so that out of a larger message we emit a new
output record only if that specific field has changed.


Since we introduced that to reduce the number of output records, our final
Kstream - KStream windowed join is not ticking anymore, although the window
is large enough (1year). Our both streams have 3 partitions.

We noticed that in the KStreams documentation, the following sentence
appears:

     * Transforming records might result in an internal data redistribution
if a key based operator (like an aggregation
     * or join) is applied to the result {@code KStream}.
     * (cf. {@link #transformValues(ValueTransformerSupplier, String...)})

What does this redistribution mean and how does it work? Why have we lost
joins result?

Thank you

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message