kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathieu D <matd...@gmail.com>
Subject kafka-streams: do not output anything while state is not stable
Date Mon, 19 Oct 2020 19:40:56 GMT
Hello there,

Let's say I need to restart my streams app from a blank state (whether by
changing app.id or using application-reset-tool).
My app is designed on "at least once" paradigm, and outputs are upserts.
The input topics have a few days worth of data, and the app will restart
from there.

If I restart it, states will be empty, meaning most outputs will be wrong.
So it should NOT upsert anything downstream for a certain period of time
(something like my biggest time window or something)

I'm thinking of a filter at the end of the topology, removing any record
whose timestamp < topic retention + biggest window size.

How do you guys manage this usually ?

Thanks for your insights

Mathieu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message