kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <thunder.stump...@gmail.com>
Subject Kafka per topic retention.bytes and global log.retention.bytes not working
Date Tue, 09 Jan 2018 18:02:08 GMT
Hello, I posted this on StackOverflow
<https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-bytes-and-global-log-retention-bytes-not-working>also
but haven't gotten any response.

thanks in advance,
Thunder
__________

We are running a 6 node cluster of kafka 0.11.0. We have set a global as
well as a per-topic retention in bytes, neither of which is being applied.
There are no errors that I can see in the logs, just nothing being deleted
(by size; the time retention does seem to be working)

See relevant configs below:

*./config/server.properties* :

# global retention 75GB or 60 days, segment size 512MB
log.retention.bytes=75000000000
log.retention.check.interval.ms=60000

log.retention.hours=1440

log.cleanup.policy=delete

log.segment.bytes=536870912

*topic configuration (30GB):*

[tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh  --zookeeper
zk-01:2181/kafka --describe --topic stg_logtopic
Topic:stg_logtopic    PartitionCount:12       ReplicationFactor:3
Configs:retention.bytes=30000000000
        Topic: stg_logtopic   Partition: 0    Leader: 4
Replicas: 4,5,6 Isr: 4,5,6
        Topic: stg_logtopic   Partition: 1    Leader: 5
Replicas: 5,6,1 Isr: 5,1,6
        ...

And, disk usage showing 910GB usage for one partition!

[tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/*
82G     /data1/kafka-data/stg_logother3-2
155G    /data1/kafka-data/stg_logother2-9
169G    /data1/kafka-data/stg_logother1-6
910G    /data1/kafka-data/stg_logtopic-4

I can see there are plenty of segment log files (512MB each) in the
partition directory... what is going on?!

Thanks in advance, Thunder

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message