kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Hays <hay...@gmail.com>
Subject retention.ms not honored for topic
Date Tue, 29 May 2018 16:01:42 GMT
A single topic does not appear to be honoring the retention.ms
setting. Three other topics (plus __consumer_offsets) on the Kafka
instance are deleting segments normally.

Kafka version: 2.12-0.10.2.1
OS: CentOS 7
Java: openjdk version "1.8.0_161"
Zookeeper: 3.4.6

Retention settings (from kafka-topics.sh describe): Topic:elk
PartitionCount:50 ReplicationFactor:2 Configs:retention.ms=7200000

Other config settings from server.properties

log.retention.hours=48
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Looking in the data directory, I see multiple segment files older than 48 hours:

-rw-r--r-- 1 root root 1073676782 May 26 20:16 00000000004713142447.log
-rw-r--r-- 1 root root 1073105605 May 26 20:18 00000000004715239774.log
-rw-r--r-- 1 root root 1072907965 May 26 20:20 00000000004717450325.log

Current date/time on server: Tue May 29 10:51:49 CDT 2018

This issue appears on all Kafka brokers and I have tried multiple
rolling restarts of all Kafka brokers and the issue remains. These
servers stopped deleting segments for this topic on May 15. This does
not correlate to any known config change. I have found no
error/warning messages in the logs to indicate a problem.

What am I missing? Thank you.

Mime
View raw message