kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koushik Chitta <kchi...@microsoft.com.INVALID>
Subject RE: Issue in retention with compact,delete cleanup policy
Date Wed, 04 Mar 2020 03:02:51 GMT
Bubbling this up to understand if anyone else are in similar use case.

-----Original Message-----
From: Koushik Chitta <kchitta@microsoft.com.INVALID> 
Sent: Sunday, February 23, 2020 1:35 PM
To: users@kafka.apache.org; dev@kafka.apache.org
Subject: [EXTERNAL] Issue in retention with compact,delete cleanup policy


I have a Topic with following config.

cleanup.policy  =  compact,delete
segment.bytes = 52428800 (~52 mb)
min.compaction.lag.ms = 1800000 (30 min) delete.retention.ms = 86400000 (1 day) retention.ms
= 259200000 (3 days)

Ideally I would want the old records > 3 days to be deleted without producing an explicit
delete(null value of a key) of the record.
But there can be a case due to continuous compaction, the segments can contain a very old
record(eg: > 30 days) and new recent record (eg: 1hr) which will make the segment ineligible
for retention delete.

Currently I don't see a work around for this. Please suggest.
I plan to start a KIP to address this use case.


View raw message