kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: questtion about log.retention.bytes
Date Tue, 20 Aug 2013 03:46:07 GMT
For the first question, yes, topic C will get the value of 30MB.

For the second question, log.retention.bytes only controls the segment log
file size, not the index. Typically, index file size is much smaller than
the log file. The index file of the last (active) segment is presized to
the max index size (defaults to 10MB). However, the size is trimmed as soon
as the segment rolls.



On Mon, Aug 19, 2013 at 9:22 AM, Paul Christian

> Hi Jun,
> Thank you for your reply. I'm still a little fuzzy on the concept.
> Are you saying I can have topic A, B and C and with
> log.retention.bytes.per.topic.A = 15MB
> log.retention.bytes.per.topic.B = 20MB
> log.retention.bytes = 30MB
> And thus topic C will get the value 30MB? Since it's not defined like the
> others' 'per topic'?
> log.retention.bytes is for all topics that are not included in
> log.retention.bytes.per.topic
> (which defines a map of topic -> size).
> Otherwise, log.retention.bytes.per.topic and log.retention.bytes seem very
> similar to me.
> Additionally, we've experimented with this value on our test cluster where
> we set the log.retention.bytes to 11MB as a test. Below is a snippet from
> our server.properties.
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> log.retention.bytes=11534336
> Here is a ls -lh from one of the topics
> -rw-r--r-- 1 kafka service  10M Aug 19 15:45 00000000000007021913.index
> -rw-r--r-- 1 kafka service 114M Aug 19 15:45 00000000000007021913.log
> The index file appears to be reflected in the property
> log.index.size.max.bytes, but the log just keeps going.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message