kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: mirroring - can we mirror a broker directly
Date Thu, 19 Apr 2012 19:49:02 GMT
The mirroring process (both the embedded consumer and the mirror-maker
tool) needs a zookeeper connection string. It should be possible to write a
mirroring tool that talks to a broker, but you would most likely then need
to have an alternate mechanism to do some load-balancing across the
mirror's consumers. In that case, you could have something as simple as
rsync'ing your partitions from each source broker to designated brokers in
your mirror cluster (I guess for that to work you would need a consistent
mapping). The easiest approach of course would be to have all your
application servers talk to a common Kafka cluster that would be an
aggregate, but you probably have a good reason to keep them separate.



> In the mirroring setup, the consumer configuration takes only zookeeper
> connection string. I am wondering whether it can point directly to a
> broker.

> Here is what I am trying to do.
> 1. run my application server and a kafka broker on the same machine. My
> application sends logs to the kafka broker
> 2. have an agggregate kafka cluster which collects logs from the kafka
> brokers on all my servers through mirroring.
> Zookeeper is not useful on my application servers since each has only one
> broker running. But I have to run it for mirroring to work. Anyway to avoid
> running zookeeper on my application server?
> Thanks,
> -Xiaoyu

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message