kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Gould <mikeyg...@gmail.com>
Subject compaction + delete not working for me
Date Fri, 06 Jan 2017 11:57:52 GMT
Hi

I'm trying to configure log compaction + deletion as per KIP-71 in kafka
0.10.1 but so far haven't had any luck. My tests show more than 50%
duplicate keys when reading from the beginning even several minutes after
all the events were sent.
The documentation in section 3.1 doesn't seem very clear to me in terms of
exactly how to configure particular behavior. Could someone please clarify
a few things for me?

In order to significantly reduce the amount of data that new subscribers
have to receive I want to compact events as soon as possible, and delete
any events more than 24 hours old (e.g if there hasn't been an update with
a matching key for 24h).

I have set

cleanup.policy=compact, delete
min.cleanable.dirty.ratio=0.5
min.compaction.lag.ms=0
retention.ms=86400000
delete.retention.ms=86460000
segment.ms=60000


   - Should the cleanup.policy be "compact,delete" or "compact, delete" or
   something else?
   - Are events eligible for compaction soon after the min.compaction.lag.ms
   time and segment.ms or is there another parameter that affects this?
   I.e. if I read from the beginning after a couple of minutes should I see no
   more than 50% of the events received have the same key as previous events.
   - Does the retention.ms parameter only affect the deletion?
   - How can I tell if the config is accepted and compaction is working? Is
   there something useful to search for in the logs?
   - Also if I change the topic config via the kafka-configs.sh tool does
   the change take effect immediately for existing events, do I have to
   restart the brokers, or does it only affect new events?

Thank you
Mike G

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message