kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Neal <cwn...@gmail.com>
Subject Re: Should MirrorMaker produce round robin with this configuration?
Date Fri, 24 Mar 2017 15:19:18 GMT
Thanks very much for the reply Manikumar!

I found that there were a few topics on the source cluster that had more
than two partitions, but all topics on the target cluster had 2
partitions.  I did a test between one topic that had 2 on each, and I did
get messages to both partitions as expected.

Is there a way to get this same behavior if the number of partitions on the
source and target topic are different?

What should I set the num.consumer.fetchers property to?  I believe I
remember reading that it needed to be at least equal to the number of
partitions on the topic.

For example, if I'm reading from:

topicA:  2 partitions
topicB:  3 partitions
topicC:  1 partition

should i set  num.consumer.fetchers to 3, or 6? :)

Again, thank you!!


On Fri, Mar 24, 2017 at 12:57 AM, Manikumar <manikumar.reddy@gmail.com>
wrote:

> Are you sure target cluster topics have more than one partition?
> If you are sending keyed messages, they may be going to the same partition.
>
> On Thu, Mar 23, 2017 at 11:15 PM, Chris Neal <cwneal@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I am using MirrorMaker to consume from a 0.8.2.2 cluster and produce to a
> > 0.10.2 cluster.  All the topics have two partitions on both clusters.  My
> > consumer.properties is:
> >
> > zookeeper.connect=[string of servers]
> > group.id=MirrorMaker
> > num.consumer.fetchers=2
> > partition.assignment.strategy=roundrobin
> >
> > My producer.properties is:
> >
> > metadata.broker.list=[string of servers]
> > compression.codec=gzip
> > producer.type=async
> > message.send.max.retries=2
> >
> > My understanding from the documentation is that this *should* work, but
> it
> > is not.  All my data is going into partition 1 on the target cluster for
> > all topics.
> >
> > Can someone help me understand what I'm missing here?
> >
> > Thank you for your time!
> > Chris
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message