kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Franz van Betteraey" <fvbetter...@web.de>
Subject Kafka Mirror Maker place of execution
Date Tue, 12 Mar 2019 13:13:06 GMT
Hi all,

there are best practices out there which recommend to run the Mirror Maker on the target cluster.
https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html

I wonder why this recommendation exists because ultimately all data must cross the border
between the clusters, regardless of whether they are consumed at the target or produced at
the source. A reason I can imagine is that the Mirror Maker supports multimple consumer but
only one producer - so consuming data on the way with the greater latency might be speed up
by the use of multiple consumers.

If performance because of multi threading is a point, would it be usefaul to use several producer
(one per consumer) to replicate the data (with a custom replication process)? Does anyone
knows why the Mirror Maker shares a single producer among all consumers?

My usecase is the replication of data from several source cluster (~10) to a single target
cluster. I would prefer to run the replication process on the source cluster to avoid to many
replication processes (each for one source) on the target cluster.

Hints and suggestions on this topic are very welcome.

Best regards
  Franz

If you would like to earn some SO recommendation points feel free to answer this question
on SO ;-)
https://stackoverflow.com/q/55122268/367285

Mime
View raw message