kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <thunder.stump...@gmail.com>
Subject Re: Kafka per topic retention.bytes and global log.retention.bytes not working
Date Tue, 09 Jan 2018 23:25:57 GMT
How would I know if we are seeing that issue? We are running 0.11.0.0 so we
would not have this fix.

On Tue, Jan 9, 2018 at 11:07 AM Wim Van Leuven <
wim.vanleuven@highestpoint.biz> wrote:

> What minor version of Kafka are you running? Might you be impacted by
> https://issues.apache.org/jira/browse/KAFKA-6030?
> -w
>
> On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges <thunder.stumpges@gmail.com>
> wrote:
>
> > Hello, I posted this on StackOverflow
> > <
> >
> https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-bytes-and-global-log-retention-bytes-not-working
> > >also
> > but haven't gotten any response.
> >
> > thanks in advance,
> > Thunder
> > __________
> >
> > We are running a 6 node cluster of kafka 0.11.0. We have set a global as
> > well as a per-topic retention in bytes, neither of which is being
> applied.
> > There are no errors that I can see in the logs, just nothing being
> deleted
> > (by size; the time retention does seem to be working)
> >
> > See relevant configs below:
> >
> > *./config/server.properties* :
> >
> > # global retention 75GB or 60 days, segment size 512MB
> > log.retention.bytes=75000000000
> > log.retention.check.interval.ms=60000
> >
> > log.retention.hours=1440
> >
> > log.cleanup.policy=delete
> >
> > log.segment.bytes=536870912
> >
> > *topic configuration (30GB):*
> >
> > [tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh  --zookeeper
> > zk-01:2181/kafka --describe --topic stg_logtopic
> > Topic:stg_logtopic    PartitionCount:12       ReplicationFactor:3
> > Configs:retention.bytes=30000000000
> >         Topic: stg_logtopic   Partition: 0    Leader: 4
> > Replicas: 4,5,6 Isr: 4,5,6
> >         Topic: stg_logtopic   Partition: 1    Leader: 5
> > Replicas: 5,6,1 Isr: 5,1,6
> >         ...
> >
> > And, disk usage showing 910GB usage for one partition!
> >
> > [tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/*
> > 82G     /data1/kafka-data/stg_logother3-2
> > 155G    /data1/kafka-data/stg_logother2-9
> > 169G    /data1/kafka-data/stg_logother1-6
> > 910G    /data1/kafka-data/stg_logtopic-4
> >
> > I can see there are plenty of segment log files (512MB each) in the
> > partition directory... what is going on?!
> >
> > Thanks in advance, Thunder
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message