kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: compaction + delete not working for me
Date Fri, 06 Jan 2017 21:36:33 GMT
On Fri, Jan 6, 2017 at 3:57 AM, Mike Gould <mikeyg123@gmail.com> wrote:

> Hi
> I'm trying to configure log compaction + deletion as per KIP-71 in kafka
> 0.10.1 but so far haven't had any luck. My tests show more than 50%
> duplicate keys when reading from the beginning even several minutes after
> all the events were sent.
> The documentation in section 3.1 doesn't seem very clear to me in terms of
> exactly how to configure particular behavior. Could someone please clarify
> a few things for me?
> In order to significantly reduce the amount of data that new subscribers
> have to receive I want to compact events as soon as possible, and delete
> any events more than 24 hours old (e.g if there hasn't been an update with
> a matching key for 24h).
> I have set
> cleanup.policy=compact, delete
> min.cleanable.dirty.ratio=0.5
> min.compaction.lag.ms=0
> retention.ms=86400000
> delete.retention.ms=86460000
> segment.ms=60000
>    - Should the cleanup.policy be "compact,delete" or "compact, delete" or
>    something else?

Either should work, extra leading and trailing spaces are removed.

>    - Are events eligible for compaction soon after the
> min.compaction.lag.ms
>    time and segment.ms or is there another parameter that affects this?

   I.e. if I read from the beginning after a couple of minutes should I see
> no
>    more than 50% of the events received have the same key as previous
> events.

Maybe you need to modify log.retention.check.interval.ms? It defaults to 5
minutes. The log cleaning runs periodically, so you may just not have
waited log enough for cleaning to have executed.

>    - Does the retention.ms parameter only affect the deletion?
>    - How can I tell if the config is accepted and compaction is working? Is
>    there something useful to search for in the logs?

Check for logs from LogCleaner.scala. It should log some info when it runs.

>    - Also if I change the topic config via the kafka-configs.sh tool does
>    the change take effect immediately for existing events, do I have to
>    restart the brokers, or does it only affect new events?

Topic config changes shouldn't need a broker restart.


> Thank you
> Mike G

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message