kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manikumar Reddy <manikumar.re...@gmail.com>
Subject Re: Compaction does not seem to kick in
Date Fri, 22 Apr 2016 03:58:57 GMT
Did you set broker config property log.cleanup.policy=compact or topic
level property cleanup.policy=compact ?

On Thu, Apr 21, 2016 at 7:16 PM, Kasim Doctor <kdoctor@expedia.com> wrote:

> Hi everyone,
>
> I have a cluster of 5 brokers with Kafka 2.10_0.8.2.1 and one of the
> topics compacted (out of a total of 4 topics).
>
> In spite of repeated attempts to tune the configuration, compaction just
> does not work. The messages that I push to this compacted topic are in the
> range of 50 MB to 90 MB.
>
> Here are my broker configs related to compaction:
>
>
>
>         log.cleaner.enable=true
>
>
>
>         log.cleaner.backoff.ms=15000
>
>
>         log.cleaner.delete.retention.ms=86400000
>
>
>         log.cleaner.threads=5
>
>
>         log.cleaner.dedupe.buffer.size=524288000
>
>
>         log.cleaner.min.cleanable.ratio=0.01
>
>
>
>
>
> Additionally, I have set segment.ms = 86400000 at the topic level for the
> specific compacted topic.
>
> I also don’t see any logs related to compaction in the kafka logs.
>
> As a result, I run of out of disk space in a matter of days since I was
> relying on compaction to free up space by deleting old entries in the log
> with the same key.
>
> Can someone please help me out here ?
>
> Thanks,
> Kasim
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message