kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Schroeder <jeffschroe...@computer.org>
Subject Re: cross-colo writing/reading?
Date Tue, 03 Mar 2015 23:34:14 GMT
Mirror maker is about separating latency and failure domains. I think it is
a very elegant solution to a difficult problem. My suspicion is that the
LinkedIn / Confluent team agrees.

On Tue, Mar 3, 2015 at 3:50 PM, Yang <teddyyyy123@gmail.com> wrote:

> thanks guys.
>
> it's just quite a lot of ops cost to setup and monitor a separate cluster,
> connected through mirror maker. sometimes if I have just a single
> producer/consumer in a new cluster, it would be more desirable to just
> connect it directly to an existing kafka setup.
>
> I remember at least in an earlier version of Cassandra (about 2 years ago,
> when I still actively worked on that), you could define network topologies
> , and define the different acknowledgement requirements for different parts
> of the topology.  if this kind of flexibility is built into the internal
> protocol of Kafka, it would be much nicer. personally I think the mirror
> maker is more of a patch work than a systematic design.
>
> On Mon, Mar 2, 2015 at 7:53 AM, Todd Palino <tpalino@gmail.com> wrote:
>
> > Latencies like this are one of the big reasons that we run our Kafka
> > clusters local to the producers and consumers. Another is network
> > partition. As Jeff noted, mirror maker is the way to connect them
> together.
> > Our architecture uses a local cluster in each datacenter, and then an
> > aggregate cluster (either in those datacenters or in another) where we
> > mirror all the traffic from the local clusters together into one place.
> > This way clients can pick a local or a global view of the data, and we
> can
> > have everyone still stay in their local Kafka clusters.
> >
> > -Todd
> >
> >
> > On Fri, Feb 27, 2015 at 4:21 PM, Yang <teddyyyy123@gmail.com> wrote:
> >
> > > we tested our new application that reads and writes to kafka.
> > >
> > > at first we found the access latency is very high. then we realized
> that
> > > it's because the client and server are in different colos. moving them
> > > together reduces down the access time to < 4 ms.
> > >
> > >
> > > I was wondering if there are any techniques/properties to set , or
> maybe
> > > enhancements to be made to the kafka code, that could cater to such
> > > situations ?
> > >
> > > Thanks
> > > Yang
> > >
> >
>



-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message