kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Diogo Vieira <diogo.vie...@kelvininc.com>
Subject Automating topic cleanup
Date Mon, 07 Jan 2019 20:02:19 GMT

I’m using a Kafka cluster with dynamically created and temporary topics that each map to
a different consumer. After a consumer disconnects the topic isn’t cleaned which means my
topics will grow with the consumers.

As far as I know, Kafka won’t automatically clean those topics (I believe something like
it was suggested as a feature when topic deletion was introduced but was never implemented).

Does anyone more knowledgeable than me can please provide some insight to tackle this problem?
I’m thinking of automating the cleanup of these topics, but I have no idea how to find out
which topics should be deleted, since I would like to have access to the timestamp of the
last consumer offset commit (or something like it) to clean topics after an arbitrary amount
of time.

Thank you in advance,
Diogo Vieira
View raw message