kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathieu Fenniak <mathieu.fenn...@replicon.com>
Subject Managing topic configuration w/ auto.create.topics.enable
Date Tue, 28 Mar 2017 16:33:53 GMT
Hey Kafka Users,

When using a Kafka broker w/ auto.create.topics.enable set to true, how do
Kafka users generally manage configuration of those topics?  In
particular, cleanup.policy={compact/delete} can be a crucial configuration
value to get correct.

In my application, I have a couple Kafka Connect sources that generate all
the input to my Kafka brokers, across about 30 different topics.  All of
these specific topics are used in a KStreams app as tables, so they should
be configured w/ cleanup.policy=compact so that state-store restoration can
take place on them without losing data.  As far as I know, there's no way
for a Kafka producer (or more specifically a KC source) to indicate topic
configuration if/when the topic is auto-created, so it needs to be done

Other topics, such as KStreams repartition topics, should be
cleanup.policy=delete, and others such as KStream changelog topics should
be cleanup.policy=compact... but KStreams handles all this for me for
internal topics.

I'm envisioning the ideal solution as something like a scheduled task that
runs against my cluster, with a configuration, that identifies and corrects
misconfigurations.  This could then roll out through different staging and
testing areas just like a general software change.  But developing such a
tool doesn't seem straight-forward since there currently isn't an API for
managing topic configuration either...


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message