kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eno Thereska <eno.there...@gmail.com>
Subject Re: compaction not happening for change log topic
Date Mon, 03 Apr 2017 12:08:41 GMT
Hi Sachin,

I’ve found this useful: http://stackoverflow.com/questions/34281643/how-to-test-whether-log-compaction-is-working-or-not-in-kafka,
<http://stackoverflow.com/questions/34281643/how-to-test-whether-log-compaction-is-working-or-not-in-kafka,as>
as well as this http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ <http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/>
(the log compaction part).

Thanks
Eno

> On Apr 1, 2017, at 6:17 AM, Sachin Mittal <sjmittal@gmail.com> wrote:
> 
> Ok I have gone through the docs and looked the log.cleaner properties.
> The use case we have is that say for a change log topic we have at a point
> of time
> (k, v1).
> When for same key we now have
> (k, v2) we really don't want (k, v1) to be retained and get cleaned up as
> soon as possible.
> 
> So I see this:
> 1. log.cleaner.delete.retention.ms 86400000
> I suppose we need to reduce this value to much less say 600000 (10 minutes
> or so from default 1 day)
> 
> 2. log.cleaner.min.compaction.lag.ms
> I think default value of 0 is OK given we want old record be clean as soon
> as possible?
> 
> 3. log.cleaner.min.cleanable.ratio
> I have not understood this one. What do we mean by dirty log?
> 
> What we have observed is that our log segment retention time for change log
> topic is 30 minutes.
> So after that time old log segment does get deleted, but what we see is all
> the segment file sizes are 1GB.
> Very rarely we see an old segment size less than 1 GB.
> So does this mean that logs are not getting compacted or the messages are
> marked for deletion but somehow log cleaner is not cleaning up those
> messages.
> 
> Please let me know is these are the properties we need to adjust for a more
> efficient cleanup of old messages for same key.
> 
> Thanks
> Sachin
> 
> 
> 
> 
> On Sat, Apr 1, 2017 at 4:54 AM, Matthias J. Sax <matthias@confluent.io>
> wrote:
> 
>> That's a topic config you need to set at the broker side:
>> 
>> See config parameter `log.cleaner.*` in
>> http://kafka.apache.org/documentation/#brokerconfigs
>> 
>> 
>> -Matthias
>> 
>> On 3/31/17 11:49 AM, Sachin Mittal wrote:
>>> Hi,
>>> I have noticed that many times change log topics don't get compacted. The
>>> segment log file is always 1 GB.
>>> 
>>> So I would like to know how and when compaction comes into play.
>>> is there a way we can get the topic compacted say trigger compaction
>> after
>>> x seconds of a given message or a given segment file creation.
>>> 
>>> Also what could be the reason for change log topic not getting compacted.
>>> 
>>> Thanks
>>> Sachin
>>> 
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message