kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Cheng <wushuja...@gmail.com>
Subject Re: Mirroring multiple clusters into one
Date Thu, 06 Jul 2017 19:37:32 GMT
I'm not sure what the "official" recommendation is. At TiVo, we *do* run all our mirrormakers
near the target cluster. It works fine for us, but we're still fairly inexperienced, so I'm
not sure how strong of a data point we should be.

I think the thought process is, if you are mirroring from a source cluster to a target cluster
where there is a WAN between the two, then whichever request goes across the WAN has a higher
chance of intermittent failure than the one over the LAN. That means that if mirrormaker is
near the source cluster, the produce request over the WAN to the target cluster may fail.
If the mirrormaker is near the target cluster, then the fetch request over the WAN to the
source cluster may fail.

Failed fetch requests don't have much impact on data replication, it just delays it. Whereas
a failure during a produce request may introduce duplicates.

Becket Qin from LinkedIn did a presentation on tuning producer performance at a meetup last
year, and I remember he specifically talked about producing over a WAN as one of the cases
where you have to tune settings. Maybe that presentation will give more ideas about what to
look at. https://www.slideshare.net/mobile/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600

-James

Sent from my iPhone

> On Jul 6, 2017, at 1:00 AM, Vahid S Hashemian <vahidhashemian@us.ibm.com> wrote:
> 
> The literature suggests running the MM on the target cluster when possible 
> (with the exception of when encryption is required for transferred data).
> I am wondering if this is still the recommended approach when mirroring 
> from multiple clusters to a single cluster (i.e. multiple MM instances).
> Is there anything in particular (metric, specification, etc.) to consider 
> before making a decision?
> 
> Thanks.
> --Vahid
> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message