kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Accessing kafka from different geo location
Date Wed, 05 Mar 2014 01:54:57 GMT
Hi Arup,

1) is hard to do since ZK stores both broker and consumer metadata, and
consumers use ZK to also discover brokers.

Not sure if I understand 2), 3) is essentially equivalent to having a
remote consumer since the KafakMirror actually have a consumer consuming
from the source (mostly remote) brokers and a producer producing to dest
(mostly local) brokers. I would recommend you first trying to have remote
consumers as is and see if the latency is acceptable.

Guozhang


On Tue, Mar 4, 2014 at 3:54 PM, Arup Malakar <amalakar@gmail.com> wrote:

> Hi,
>
> We have a use case where we have the kafka cluster in a different
> datacenter from the consumers. The kafka cluster is in the same data center
> where producers are. I have heard that zookeeper could be slow if the
> zookeper clients are in different geo location. Is there any best practice
> for this kind of scenario?
>
> I can think of following options:
>
> 1. If kafka supports consumers wherein consumers can talk to a different
> zookeeper cluster than the ones which the kafka cluster talks to. This
> assumes that consumers accesses zookeeper only to store consumer group
> metadata, and it obtains broker related metadata from kafka broker itself.
> This would let us have a zookeeper cluster in the same datacenter as the
> consumers. Does kafka architecture allow this at all?
>
> 2. Have a zookeeper observer in the consumer datacenter.
>
> 3. Have a kafka cluster in the consumer datacenter and use KafkaMirror to
> mirror between the two datacenters.
>
> 4. Move kafka cluster from producer data center to consumer datacenter.
> This would mean the producers would have to write to kafka across data
> centers.
>
> I would highly appreciate your thoughts/suggestions.
>
> Thanks,
> Arup Malakar
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message