kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xiaoyu wang <xiaoyu.w...@gmail.com>
Subject Re: can I start a kafka server that mirrors more than one cluster?
Date Wed, 18 Apr 2012 20:19:48 GMT
Thanks, Joey,  filed jira

https://issues.apache.org/jira/browse/KAFKA-333

On Wed, Apr 18, 2012 at 12:55 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> > I am trying out kafka mirroring. It seems to me that I can start a kafka
> > server (broker) to mirror one cluster only. So in order to mirror N
> source
> > clusters to one aggregate cluster, I will need to start N brokers with
> each
> > mirror one source cluster. Is this correct?
> >
>
> That's right. Also, you may want to take a look at the new mirroring
> mechanism which will help address your scenario. It was recently added (in
> trunk) so there's not much by way of documentation apart from the code
> itself. It is pretty simple:
> - Start up your mirror cluster
> - Run kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> cluster1_consumer.properties --consumer.config cluster2_consumer.properties
> --producer.config mirror_producer.properties --whitelist=".*"
>
> I'll update that mirroring wiki soon, since the embedded consumer approach
> is being deprecated in favor of the stand-alone mirroring tool.
>
>
> > Also, is there a plan to enable non-random partitioning in the build-in
> > producer for mirroring?
> >
> >
> This would be a good feature to have - can you file a jira to track it? I'd
> have to think through how best to implement it (right now the partitioning
> key type needs to be embedded in the code. The mirroring tool instantiates
> a producer directly and assumes random partitioning.)
>
>


> Thanks,
>
> Joel
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message