kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wim Van Leuven (BB.io)" <wim.vanleu...@bigboards.io>
Subject 1 topic that keeps growing
Date Tue, 12 Sep 2017 19:39:38 GMT
Hello,

we have a 3 node kafka cluster setup, with quite a bunch of topics, that
have a nice life, regular cleans, compact, etc. 1 topic however keeps
growing, new segments are recreated at regular intervals, but old segments
are never deleted.

Our setup is based on Confluent 3.2.0 OSS, hence Apache Kafka 0.10.2.0. The
server.props related to log retention are fairly straightforward:

log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.check.interval.ms=300000
log.retention.bytes=-1
log.retention.ms=7200000
log.segment.bytes=1073741824

We do apply specific configurations for each topic to ensure proper
lifetime, cleanup etc. For the topic at hand, those are

retention.ms=3600001
segment.ms=3600000
compression.type=uncompressed
cleanup.policy=delete

I tried quite a bunch of configurations to try various setups, but I'm in a
dead end and my final resort would be to delete the topic (and all data
underneath) and just recreate. I still can because this is a test/staging
environment. I'm a bit reluctant as I'd like to understand what's going on
because if this hits us on a production system ...

Any help or insights would be much appreciated!
-wim

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message