kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: can I start a kafka server that mirrors more than one cluster?
Date Wed, 18 Apr 2012 20:46:58 GMT
Producer actually supports partitioning by a partitioning key. So you just
need to select a partitioner (default is partitioning based on the hash
value of the key) and put in a key in ProduceData.

Thanks,

Jun

On Wed, Apr 18, 2012 at 1:19 PM, xiaoyu wang <xiaoyu.wang@gmail.com> wrote:

> Thanks, Joey,  filed jira
>
> https://issues.apache.org/jira/browse/KAFKA-333
>
> On Wed, Apr 18, 2012 at 12:55 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:
>
> > > I am trying out kafka mirroring. It seems to me that I can start a
> kafka
> > > server (broker) to mirror one cluster only. So in order to mirror N
> > source
> > > clusters to one aggregate cluster, I will need to start N brokers with
> > each
> > > mirror one source cluster. Is this correct?
> > >
> >
> > That's right. Also, you may want to take a look at the new mirroring
> > mechanism which will help address your scenario. It was recently added
> (in
> > trunk) so there's not much by way of documentation apart from the code
> > itself. It is pretty simple:
> > - Start up your mirror cluster
> > - Run kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > cluster1_consumer.properties --consumer.config
> cluster2_consumer.properties
> > --producer.config mirror_producer.properties --whitelist=".*"
> >
> > I'll update that mirroring wiki soon, since the embedded consumer
> approach
> > is being deprecated in favor of the stand-alone mirroring tool.
> >
> >
> > > Also, is there a plan to enable non-random partitioning in the build-in
> > > producer for mirroring?
> > >
> > >
> > This would be a good feature to have - can you file a jira to track it?
> I'd
> > have to think through how best to implement it (right now the
> partitioning
> > key type needs to be embedded in the code. The mirroring tool
> instantiates
> > a producer directly and assumes random partitioning.)
> >
> >
>
>
> > Thanks,
> >
> > Joel
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message