kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kopacki, Tomasz (Nokia - PL/Wroclaw)" <tomasz.kopa...@nokia.com>
Subject log compaction v log rotation - best of the two worlds
Date Wed, 21 Mar 2018 09:06:31 GMT
Hi,
I've been recently exploring log handling in kafka and I wonder if/how can I mixed log compaction
with log rotation.
A little background first:
I have an application that uses kafka topics as a backend for event sourcing. Messages represents
change of state of my 'resources'. Each resource has UID that is also used as a key for the
messages.
My resources have a lifecycle and when their life ends I don't need them anymore and there
is no point in keeping their history. Having said that I thought that best choice for me will
be log compaction with tombstone feature but I also would like to have a possibility to keep
history of changes for the resources(only until they die).
With those requirements I'd love to have a possibility to use tombstone feature for log rotation
but I guess it ain't working like that.

Does anyone here had similar requirements and solve that somehow ?


Sincerely,
Tomasz Kopacki
DevOps Engineer @ Nokia


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message