kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Cheng <wushuja...@gmail.com>
Subject When using mirrormaker, how are people creating topics?
Date Mon, 05 Dec 2016 08:42:10 GMT

We are using mirrormaker to mirror topics from one cluster to another, and I wanted to get
some advice from the community on how people are doing mirroring. In particular, how are people
dealing with topic creation?

Do you turn on auto-topic creation in your destination clusters (auto.create.topics.enable=true)?

If not, do you manually create the individual destination topics?

If so, how does that work with mirroring based on a whitelist (regex)?

The way we are doing it right now is, we have our regex in a file somewhere. The regex is
used in 2 ways:
1) Passed to mirrormaker, to do the mirroring.
2) Passed to a program which looks up all the topics on the source cluster, finds the ones
that match the regex, and then creates them on the destination cluster. (We have auto-topic
creation turned off auto.create.topics.enable=false)

One downside of that approach is there a potential race, where if the regex changes, then
mirrormaker (in #1) might start trying to produce to a new destination topic before the topic
was created (by #2).

Some other hand-wavy ideas that came to mind might be:
* handling topic creation in a MirrorMakerMessageHandler
* handling topic creation in an interceptor

Anyway, was hoping to get some thoughts from people who are already doing this.


View raw message