kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Neal <cwn...@gmail.com>
Subject Re: Help with Mirror Maker config to produce to all partitions in target cluster
Date Thu, 02 Nov 2017 18:04:02 GMT
Thanks Manikumar for the reply :)

For errors in the logs, I have a few WARN messages about "Reconnect due to
socked error: null", and an occasional "Connection timed out", but that's
it.

I definitely can turn on trace logs.  Is there something specific you know
of that I should look for?

WRT keys, all 10 partitions are used on the source cluster, so doesn't that
indicate that the keys are different?  I don't produce the data, so I'm not
100% sure, but I can try and get ahold of the producer's source code and
see how they are being created (if at all).

I did validate that the partition counts are the same in both clusters.

I was trying to determine if I had the right values set for num.streams
and num.consumer.fetchers based on consuming from/producing to 267 topics
with 10 partitions each...or if that even matters in this case.

Chris


On Thu, Nov 2, 2017 at 12:58 AM, Manikumar <manikumar.reddy@gmail.com>
wrote:

> Any exceptions in the mirror maker logs? may be you can enable mirror maker
> trace logs.
> maybe all messages are having same key? Can you recheck partition count on
> target cluster?
>
>
> On Thu, Nov 2, 2017 at 2:45 AM, Chris Neal <cwneal@gmail.com> wrote:
>
> > Apologies for bumping my own post, but really hoping someone has
> experience
> > with MirrorMaker who might be able to help with my question below. :)
> >
> > Thanks!
> > Chris
> >
> > On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cwneal@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I've been struggling with this for awhile and need some help please. :)
> > >
> > > I have a source cluster running Kafka 0.8.1.1 with about 900 topics.
> > > I have a target cluster running Kafka 0.10.2 with those same topics
> > > pre-created with the same number of partitions per topic as the source
> > > cluster.
> > >
> > > I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot
> seem
> > > to get the produced records to go anywhere besides partition 0 of all
> > these
> > > topics. :(
> > >
> > > I found this page to be helpful WRT performance tuning, but it did not
> > > solve my partition producer problem:
> > >
> > > https://community.hortonworks.com/articles/79891/kafka-
> > > mirror-maker-best-practices.html
> > >
> > > My consumer.config:
> > > ======================================
> > > zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
> > > group.id=prod_perf_mirrormaker2_new1
> > > num.consumer.fetchers=1100
> > > client.id=MirrorMakerConsumer_perf
> > > ======================================
> > >
> > > My producer config:
> > > ======================================
> > > metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092
> > > compression.codec=gzip
> > > producer.type=async
> > > batch.num.messages=1000
> > > message.send.max.retries=2
> > > client.id=MirrorMakerProducer_perf
> > > ======================================
> > >
> > > My start script:
> > > ======================================
> > > #!/bin/sh
> > >
> > > export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
> > >
> > > /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh
> > > kafka.tools.MirrorMaker \
> > > --producer.config /home/chris.neal/mirror_maker/
> > prod_perf/producer.properties
> > > \
> > > --consumer.config /home/chris.neal/mirror_maker/
> > prod_perf/consumer.properties
> > > \
> > > --num.streams=10 \
> > > --whitelist '<snip long list of about 60 topics>'
> > > ======================================
> > >
> > > I do know that there is some combination of "num.streams" and
> > > "num.consumer.fetchers" that should get it working, but I can't figure
> > out
> > > what that combination is.
> > >
> > > All topics have the same setup of 10 partitions per topic.
> > >
> > > Can anyone shed some light on how to make this go across all 10
> > > partitions, and not just partition 0?
> > >
> > > Thank you SO much for your time and help.
> > > Chris
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message