kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wim Van Leuven <wim.vanleu...@highestpoint.biz>
Subject Delayed processing
Date Thu, 08 Mar 2018 20:35:13 GMT

I'm wondering how to design a KStreams or regular Kafka application that
can hold of processing of messages until a future time.

This related to EU's data protection regulation: we can store raw messages
for a given time; afterwards we have to store the anonymised message. So, I
was thinking about branching the stream, anonymise the messages into a
waiting topic and than continue from there until the retention time passes.

But that approach has some caveats:

   - This is not an exact solution as order of events is not guaranteed: we
   might encounter a message that triggers the stop processing while some
   events arriving later should normally still pass
   - how to stop properly stop processing if we encounter a message that
   indicates to not continue?
   - ...

Are there better know solutions or best practices to delay message
processing with Kafka streams / consumers+producers?

Thanks for any insights/help here!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message