kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Mittal <sjmit...@gmail.com>
Subject Re: compaction not happening for change log topic
Date Sat, 01 Apr 2017 05:17:58 GMT
Ok I have gone through the docs and looked the log.cleaner properties.
The use case we have is that say for a change log topic we have at a point
of time
(k, v1).
When for same key we now have
(k, v2) we really don't want (k, v1) to be retained and get cleaned up as
soon as possible.

So I see this:
1. log.cleaner.delete.retention.ms 86400000
I suppose we need to reduce this value to much less say 600000 (10 minutes
or so from default 1 day)

2. log.cleaner.min.compaction.lag.ms
I think default value of 0 is OK given we want old record be clean as soon
as possible?

3. log.cleaner.min.cleanable.ratio
I have not understood this one. What do we mean by dirty log?

What we have observed is that our log segment retention time for change log
topic is 30 minutes.
So after that time old log segment does get deleted, but what we see is all
the segment file sizes are 1GB.
Very rarely we see an old segment size less than 1 GB.
So does this mean that logs are not getting compacted or the messages are
marked for deletion but somehow log cleaner is not cleaning up those

Please let me know is these are the properties we need to adjust for a more
efficient cleanup of old messages for same key.


On Sat, Apr 1, 2017 at 4:54 AM, Matthias J. Sax <matthias@confluent.io>

> That's a topic config you need to set at the broker side:
> See config parameter `log.cleaner.*` in
> http://kafka.apache.org/documentation/#brokerconfigs
> -Matthias
> On 3/31/17 11:49 AM, Sachin Mittal wrote:
> > Hi,
> > I have noticed that many times change log topics don't get compacted. The
> > segment log file is always 1 GB.
> >
> > So I would like to know how and when compaction comes into play.
> > is there a way we can get the topic compacted say trigger compaction
> after
> > x seconds of a given message or a given segment file creation.
> >
> > Also what could be the reason for change log topic not getting compacted.
> >
> > Thanks
> > Sachin
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message