kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wim Van Leuven <wim.vanleu...@highestpoint.biz>
Subject Re: Kafka per topic retention.bytes and global log.retention.bytes not working
Date Tue, 09 Jan 2018 19:06:48 GMT
What minor version of Kafka are you running? Might you be impacted by
https://issues.apache.org/jira/browse/KAFKA-6030?
-w

On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges <thunder.stumpges@gmail.com>
wrote:

> Hello, I posted this on StackOverflow
> <
> https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-bytes-and-global-log-retention-bytes-not-working
> >also
> but haven't gotten any response.
>
> thanks in advance,
> Thunder
> __________
>
> We are running a 6 node cluster of kafka 0.11.0. We have set a global as
> well as a per-topic retention in bytes, neither of which is being applied.
> There are no errors that I can see in the logs, just nothing being deleted
> (by size; the time retention does seem to be working)
>
> See relevant configs below:
>
> *./config/server.properties* :
>
> # global retention 75GB or 60 days, segment size 512MB
> log.retention.bytes=75000000000
> log.retention.check.interval.ms=60000
>
> log.retention.hours=1440
>
> log.cleanup.policy=delete
>
> log.segment.bytes=536870912
>
> *topic configuration (30GB):*
>
> [tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh  --zookeeper
> zk-01:2181/kafka --describe --topic stg_logtopic
> Topic:stg_logtopic    PartitionCount:12       ReplicationFactor:3
> Configs:retention.bytes=30000000000
>         Topic: stg_logtopic   Partition: 0    Leader: 4
> Replicas: 4,5,6 Isr: 4,5,6
>         Topic: stg_logtopic   Partition: 1    Leader: 5
> Replicas: 5,6,1 Isr: 5,1,6
>         ...
>
> And, disk usage showing 910GB usage for one partition!
>
> [tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/*
> 82G     /data1/kafka-data/stg_logother3-2
> 155G    /data1/kafka-data/stg_logother2-9
> 169G    /data1/kafka-data/stg_logother1-6
> 910G    /data1/kafka-data/stg_logtopic-4
>
> I can see there are plenty of segment log files (512MB each) in the
> partition directory... what is going on?!
>
> Thanks in advance, Thunder
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message