kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kasim Doctor <kdoc...@expedia.com>
Subject Compaction does not seem to kick in
Date Thu, 21 Apr 2016 13:46:37 GMT
Hi everyone,

I have a cluster of 5 brokers with Kafka 2.10_0.8.2.1 and one of the topics compacted (out
of a total of 4 topics).

In spite of repeated attempts to tune the configuration, compaction just does not work. The
messages that I push to this compacted topic are in the range of 50 MB to 90 MB.

Here are my broker configs related to compaction:



        log.cleaner.enable=true



        log.cleaner.backoff.ms=15000


        log.cleaner.delete.retention.ms=86400000


        log.cleaner.threads=5


        log.cleaner.dedupe.buffer.size=524288000


        log.cleaner.min.cleanable.ratio=0.01





Additionally, I have set segment.ms = 86400000 at the topic level for the specific compacted
topic.

I also don’t see any logs related to compaction in the kafka logs.

As a result, I run of out of disk space in a matter of days since I was relying on compaction
to free up space by deleting old entries in the log with the same key.

Can someone please help me out here ?

Thanks,
Kasim
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message