kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <thunder.stump...@gmail.com>
Subject Kafka per topic retention.bytes and global log.retention.bytes not working
Date Tue, 09 Jan 2018 18:02:08 GMT
Hello, I posted this on StackOverflow
but haven't gotten any response.

thanks in advance,

We are running a 6 node cluster of kafka 0.11.0. We have set a global as
well as a per-topic retention in bytes, neither of which is being applied.
There are no errors that I can see in the logs, just nothing being deleted
(by size; the time retention does seem to be working)

See relevant configs below:

*./config/server.properties* :

# global retention 75GB or 60 days, segment size 512MB




*topic configuration (30GB):*

[tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh  --zookeeper
zk-01:2181/kafka --describe --topic stg_logtopic
Topic:stg_logtopic    PartitionCount:12       ReplicationFactor:3
        Topic: stg_logtopic   Partition: 0    Leader: 4
Replicas: 4,5,6 Isr: 4,5,6
        Topic: stg_logtopic   Partition: 1    Leader: 5
Replicas: 5,6,1 Isr: 5,1,6

And, disk usage showing 910GB usage for one partition!

[tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/*
82G     /data1/kafka-data/stg_logother3-2
155G    /data1/kafka-data/stg_logother2-9
169G    /data1/kafka-data/stg_logother1-6
910G    /data1/kafka-data/stg_logtopic-4

I can see there are plenty of segment log files (512MB each) in the
partition directory... what is going on?!

Thanks in advance, Thunder

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message