kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: log retention and rollover
Date Mon, 02 Jun 2014 23:22:42 GMT

We initially set the rolling criterion based on hours to avoid too frequent
log rolling and in turn too small segment files. For your case this may be
reasonable to set the rolling criterion on minutes. Could you file a JIRA?


On Mon, Jun 2, 2014 at 4:00 PM, Steven Wu <stevenwu@netflix.com.invalid>

> This might be a bit unusual. We have a topic that we only need to keep last
> 5 minutes of msgs so that replay from beginning is fast.
> Although retention.ms has time unit of minute, segment.ms ONLY has time
> unit of hour. If I understand cleanup correctly, it can only delete files
> that are rolled over. If true, the minimal retention period can be actually
> one hour.
> is there any particular reason for different time units for retention and
> roll? Can we add "log.roll.minutes"?
> retention.ms7 dayslog.retention.minutesThis configuration controls the
> maximum time we will retain a log before we will discard old log segments
> to free up space if we are using the "delete" retention policy. This
> represents an SLA on how soon consumers must read their data.
> segment.ms7 dayslog.roll.hoursThis configuration controls the period of
> time after which Kafka will force the log to roll even if the segment file
> isn't full to ensure that retention can delete or compact old data.
> Thanks,
> Steven

-- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message