kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Fouché <nfou...@onfocus.io>
Subject Dismissing late messages in Kafka Streams
Date Thu, 20 Oct 2016 07:40:12 GMT
Hi,

I aggregate some data with `aggregateByKey` and a `TimeWindows`.

I set the maintain duration of the window to 30 days.
If it consumes a message older than 30 days, then a new aggregate is
created for this old window.
The problem is that this old windowed aggregate is of course incomplete and
will overwrite a record in the final database.

So is there a way to dismiss these old messages ?

I only see the point of accepting old messages when the topology is
launched in "repair" mode.
By the way, is there any article about replaying old messages. Some tips
and tricks, like "you'd better do that in another deployment of your
topology", and/or "you'd better use topics dedicated to repair".

Thanks
Nicolas

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message