kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kasim Doctor <kdoc...@expedia.com>
Subject Re: Compaction does not seem to kick in
Date Tue, 26 Apr 2016 13:07:17 GMT
Yes, we have indeed set the cleanup.policy=compact for the specific topic.




On 4/21/16, 11:58 PM, "Manikumar Reddy" <manikumar.reddy@gmail.com> wrote:

>Did you set broker config property log.cleanup.policy=compact or topic
>level property cleanup.policy=compact ?
>
>On Thu, Apr 21, 2016 at 7:16 PM, Kasim Doctor <kdoctor@expedia.com> wrote:
>
>> Hi everyone,
>>
>> I have a cluster of 5 brokers with Kafka 2.10_0.8.2.1 and one of the
>> topics compacted (out of a total of 4 topics).
>>
>> In spite of repeated attempts to tune the configuration, compaction just
>> does not work. The messages that I push to this compacted topic are in the
>> range of 50 MB to 90 MB.
>>
>> Here are my broker configs related to compaction:
>>
>>
>>
>>         log.cleaner.enable=true
>>
>>
>>
>>         log.cleaner.backoff.ms=15000
>>
>>
>>         log.cleaner.delete.retention.ms=86400000
>>
>>
>>         log.cleaner.threads=5
>>
>>
>>         log.cleaner.dedupe.buffer.size=524288000
>>
>>
>>         log.cleaner.min.cleanable.ratio=0.01
>>
>>
>>
>>
>>
>> Additionally, I have set segment.ms = 86400000 at the topic level for the
>> specific compacted topic.
>>
>> I also don’t see any logs related to compaction in the kafka logs.
>>
>> As a result, I run of out of disk space in a matter of days since I was
>> relying on compaction to free up space by deleting old entries in the log
>> with the same key.
>>
>> Can someone please help me out here ?
>>
>> Thanks,
>> Kasim
>>
Mime
View raw message