kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arup Malakar <amala...@gmail.com>
Subject Re: Accessing kafka from different geo location
Date Wed, 05 Mar 2014 03:54:35 GMT
Thanks for the reply Guozhang. See inline comments:


On Tue, Mar 4, 2014 at 5:54 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> 1) is hard to do since ZK stores both broker and consumer metadata, and
> consumers use ZK to also discover brokers.
>

Thanks for the confirmation.

Not sure if I understand 2),


The concern of connecting to zookeeper in a different datacenter is that
zookeeper is very sensitive to network latencies.
If heartbeats are not sent by the client within the timeout period the
session expires. So the client would require to reconnect, this was my main
concern of consumign from a kafka cluster across data centers. Zookeeper
has a notion of observer node, which acts as a bridge between datacenters
for zookeeper. In the use case mentioned I could have a zookeeper observer
node in the same datacenter node as the consumers.

More: http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html

I was wondering if anyone has used zookeeper observer node for kafka
consumers and if there is any downside to it.


> 3) is essentially equivalent to having a
> remote consumer since the KafakMirror actually have a consumer consuming
> from the source (mostly remote) brokers and a producer producing to dest
> (mostly local) brokers. I would recommend you first trying to have remote
> consumers as is and see if the latency is acceptable.
>

Does MirrorMaker have same level of interaction with zookeeper as consumer
groups? If yes then you are right, using MirrorMaker would be same as
having remote consumers. Yes I will check the latency of consuming from a
different datacenter acceptable or not for our usecase.

Thanks,
Arup

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message