kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: can I start a kafka server that mirrors more than one cluster?
Date Wed, 18 Apr 2012 19:55:40 GMT
> I am trying out kafka mirroring. It seems to me that I can start a kafka
> server (broker) to mirror one cluster only. So in order to mirror N source
> clusters to one aggregate cluster, I will need to start N brokers with each
> mirror one source cluster. Is this correct?
>

That's right. Also, you may want to take a look at the new mirroring
mechanism which will help address your scenario. It was recently added (in
trunk) so there's not much by way of documentation apart from the code
itself. It is pretty simple:
- Start up your mirror cluster
- Run kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
cluster1_consumer.properties --consumer.config cluster2_consumer.properties
--producer.config mirror_producer.properties --whitelist=".*"

I'll update that mirroring wiki soon, since the embedded consumer approach
is being deprecated in favor of the stand-alone mirroring tool.


> Also, is there a plan to enable non-random partitioning in the build-in
> producer for mirroring?
>
>
This would be a good feature to have - can you file a jira to track it? I'd
have to think through how best to implement it (right now the partitioning
key type needs to be embedded in the code. The mirroring tool instantiates
a producer directly and assumes random partitioning.)

Thanks,

Joel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message