kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Issue in retention with compact,delete cleanup policy
Date Wed, 04 Mar 2020 04:35:13 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

If I understand the issue correctly, the problem is that some data,
even if it is not updated for a long period of time will, never be
deleted because it's always in a segment with new data?

As your segment size is already fairly small, I don't see any other
solution as to write corresponding tombstones.

Changing the behavior would for sure require a KIP. One could
basically change the compaction to leave data that is older than the
retention time in their own segments to make them available for
deletion. I am not familiar with this part of the code and thus cannot
say how complex it would be to implement.


- -Matthias


On 3/3/20 7:02 PM, Koushik Chitta wrote:
> Bubbling this up to understand if anyone else are in similar use
> case.
>
>
> -----Original Message----- From: Koushik Chitta
> <kchitta@microsoft.com.INVALID> Sent: Sunday, February 23, 2020
> 1:35 PM To: users@kafka.apache.org; dev@kafka.apache.org Subject:
> [EXTERNAL] Issue in retention with compact,delete cleanup policy
>
> Hi,
>
> I have a Topic with following config.
>
> cleanup.policy  =  compact,delete segment.bytes = 52428800 (~52
> mb) min.compaction.lag.ms = 1800000 (30 min) delete.retention.ms =
> 86400000 (1 day) retention.ms = 259200000 (3 days)
>
> Ideally I would want the old records > 3 days to be deleted without
> producing an explicit delete(null value of a key) of the record.
> But there can be a case due to continuous compaction, the segments
> can contain a very old record(eg: > 30 days) and new recent record
> (eg: 1hr) which will make the segment ineligible for retention
> delete.
>
> Currently I don't see a work around for this. Please suggest. I
> plan to start a KIP to address this use case.
>
> Thanks, Koushik
>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5fMAEACgkQO4miYXKq
/OgBpA//auSjqG8bnpKD44Svey2GK7cI1kA6pyEY/NMgS4PLav5q+dWiPnADBDQV
ZmgekEcXLk6TRggl8oHs0zJCf9ETesGwAUQEcQnIstK+lPIBT/fOEpVdJiZzwNt5
24t1flhO3NBgFty+XQm5J0DrNJmMaysbhptuulPpOfn2Cqj6L26Co22rWkm1w9Pa
03bmrswj5f2kVXWvKExY1kZLRxNhGu4fQaovyDF8dgTZAoHQWAwXpUOxB70tl0Ct
fDwzzYd+waGNnQ7cbsbdY6QxvhQJj2aijXCgih3tqJ2ww14BIlTHBku9FAIU77Ww
OesfAFAt8tcZmYX5pntVyaG2FblATDKSnf1WEAzSRNzuw+dGsnrkAZSwizZJWVYq
JgRo3EO55HHPDqol+T1jI/KFL7tUb549rEd6Er3sxM/+PHX7CeKoS+Y/+VOPY8Z3
uc6qVe1y+rjqikBE9dmOjBmJngopbYtAK8Wu6CbdNRqDXXZtkXsnoj1vr7SlsJZ1
yov+11gLEAtGHSrAwGw6W5Jydcwdbxug2xSgWI0W0XD6djiyS3NBw59vBfB4+wPP
YLrS5UvxB3W6/wUgivx3iPNTOtHMuqb4lElxVLgHQzKJUmQkouDGQTpUNKCEfncb
9631uJP9hAMLOl/rvPW6CDp2PlMZLAgB14vIgqyEB0vkU7D+1qs=
=YTFS
-----END PGP SIGNATURE-----

Mime
View raw message