kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathieu D <matd...@gmail.com>
Subject Re: kafka-streams: do not output anything while state is not stable
Date Fri, 23 Oct 2020 18:04:34 GMT
Ok. Thanks ;-)

Le mar. 20 oct. 2020 à 19:12, Matthias J. Sax <mjsax@apache.org> a écrit :

> It's highly use-case dependent, but applying a filter at the end does
> sound like a good solution to me.
>
> -Matthias
>
> On 10/19/20 12:40 PM, Mathieu D wrote:
> > Hello there,
> >
> > Let's say I need to restart my streams app from a blank state (whether by
> > changing app.id or using application-reset-tool).
> > My app is designed on "at least once" paradigm, and outputs are upserts.
> > The input topics have a few days worth of data, and the app will restart
> > from there.
> >
> > If I restart it, states will be empty, meaning most outputs will be
> wrong.
> > So it should NOT upsert anything downstream for a certain period of time
> > (something like my biggest time window or something)
> >
> > I'm thinking of a filter at the end of the topology, removing any record
> > whose timestamp < topic retention + biggest window size.
> >
> > How do you guys manage this usually ?
> >
> > Thanks for your insights
> >
> > Mathieu
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message