kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <steve...@netflix.com.INVALID>
Subject log retention and rollover
Date Mon, 02 Jun 2014 23:00:33 GMT
This might be a bit unusual. We have a topic that we only need to keep last
5 minutes of msgs so that replay from beginning is fast.

Although retention.ms has time unit of minute, segment.ms ONLY has time
unit of hour. If I understand cleanup correctly, it can only delete files
that are rolled over. If true, the minimal retention period can be actually
one hour.

is there any particular reason for different time units for retention and
roll? Can we add "log.roll.minutes"?

retention.ms7 dayslog.retention.minutesThis configuration controls the
maximum time we will retain a log before we will discard old log segments
to free up space if we are using the "delete" retention policy. This
represents an SLA on how soon consumers must read their data.

segment.ms7 dayslog.roll.hoursThis configuration controls the period of
time after which Kafka will force the log to roll even if the segment file
isn't full to ensure that retention can delete or compact old data.

Thanks,
Steven

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message