kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Neal <cwn...@gmail.com>
Subject Re: Should MirrorMaker produce round robin with this configuration?
Date Fri, 24 Mar 2017 19:48:42 GMT
lol...well, I take it all back.  Now I can't get it to work at all :(

Here's what I have:

*consumer.properties*
zookeeper.connect=[server_list]
# I was changing the group.id each time in case that was causing some issues
group.id=MirrorMakerTest7
client.id=MirrorMakerConsumer

*producer.properties*
metadata.broker.list=[server_list]
compression.codec=gzip
producer.type=async
batch.num.messages=100
message.send.max.retries=2
client.id=MirrorMakerProducer

*script:*
#!/bin/sh

export KAFKA_HEAP_OPTS="-Xmx32M -Xms32M"

/home/chris.neal/kafka_2.10-0.8.1.1/bin/kafka-run-class.sh
kafka.tools.MirrorMaker \
--producer.config /home/chris.neal/mirror_maker/test3/producer.properties \
--consumer.config /home/chris.neal/mirror_maker/test3/consumer.properties \
--num.streams=2 \
--num.producers=2 \
--whitelist 'mytopic'

The topic definition is as follows:

*Source cluster:*
Topic: mytopic PartitionCount:2 ReplicationFactor:2 Configs:
Topic: mytopic Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: mytopic Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1

*Target cluster:*
Topic: mytopic PartitionCount:2 ReplicationFactor:2 Configs:
Topic: mytopic Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: mytopic Partition: 1 Leader: 2 Replicas: 2,3 Isr: 2,3

The target has one more replica (3 servers in cluster vs. 2), but that's
the only difference.  I've spent seriously all day trying various configs,
and I can't get this simple thing working...

Sorry to keep bugging you for something that should be quite simple.  :(
Chris



On Fri, Mar 24, 2017 at 12:37 PM, Chris Neal <cwneal@gmail.com> wrote:

> Again, thank you for the feedback.  That link was very helpful!
>
> I adjusted my consumer/producer configs to be:
>
> consumer:
> zookeeper.connect=[server_list_here]
> group.id=MirrorMaker
> exclude.internal.topics=true
> client.id=MirrorMakerConsumer
>
> producer:
> metadata.broker.list=[server_list_here]
> compression.codec=gzip
> producer.type=async
> batch.size=100
> message.send.max.retries=2
> client.id=MirrorMakerProducer
>
> script file:
> /home/chris.neal/kafka_2.10-0.8.1.1/bin/kafka-run-class.sh
> kafka.tools.MirrorMaker \
> --producer.config /home/chris.neal/mirror_maker/producer.properties \
> --consumer.config /home/chris.neal/mirror_maker/consumer.properties \
> --num.streams=18 \
> --num.producers=18 \
> --whitelist '[topic_list_here]'
>
> The num.streams matches the *total* of all partitions for the topics I am
> consuming.  The topics on both the source/target cluster have the same
> partitions as well.  I did have to adjust one to make this true.
>
> But...The messages still all go to partition 1 for all the topics. :(
>
> The only time I have been able to get it to round-robin is when I pick
> just one topic.  That really makes me think it is still a configuration
> issue that is causing my problems (probably num.streams).
>
> Thanks again for working through this with me.  VERY much appreciate your
> help.
> Chris
>
>
> On Fri, Mar 24, 2017 at 10:48 AM, Manikumar <manikumar.reddy@gmail.com>
> wrote:
>
>> producer distributes the non-keyed messages to available target partitions
>> in a
>> round-robin fashion.
>>
>> You don't need to set num.consumer.fetchers, partition.assignment.strategy
>> props.
>> Use the --num.streams option to specify the number of consumer threads to
>> create.
>>
>> https://community.hortonworks.com/articles/79891/kafka-mirro
>> r-maker-best-practices.html
>>
>> On Fri, Mar 24, 2017 at 8:49 PM, Chris Neal <cwneal@gmail.com> wrote:
>>
>> > Thanks very much for the reply Manikumar!
>> >
>> > I found that there were a few topics on the source cluster that had more
>> > than two partitions, but all topics on the target cluster had 2
>> > partitions.  I did a test between one topic that had 2 on each, and I
>> did
>> > get messages to both partitions as expected.
>> >
>> > Is there a way to get this same behavior if the number of partitions on
>> the
>> > source and target topic are different?
>> >
>> > What should I set the num.consumer.fetchers property to?  I believe I
>> > remember reading that it needed to be at least equal to the number of
>> > partitions on the topic.
>> >
>> > For example, if I'm reading from:
>> >
>> > topicA:  2 partitions
>> > topicB:  3 partitions
>> > topicC:  1 partition
>> >
>> > should i set  num.consumer.fetchers to 3, or 6? :)
>> >
>> > Again, thank you!!
>> >
>> >
>> > On Fri, Mar 24, 2017 at 12:57 AM, Manikumar <manikumar.reddy@gmail.com>
>> > wrote:
>> >
>> > > Are you sure target cluster topics have more than one partition?
>> > > If you are sending keyed messages, they may be going to the same
>> > partition.
>> > >
>> > > On Thu, Mar 23, 2017 at 11:15 PM, Chris Neal <cwneal@gmail.com>
>> wrote:
>> > >
>> > > > Hi everyone,
>> > > >
>> > > > I am using MirrorMaker to consume from a 0.8.2.2 cluster and produce
>> > to a
>> > > > 0.10.2 cluster.  All the topics have two partitions on both
>> clusters.
>> > My
>> > > > consumer.properties is:
>> > > >
>> > > > zookeeper.connect=[string of servers]
>> > > > group.id=MirrorMaker
>> > > > num.consumer.fetchers=2
>> > > > partition.assignment.strategy=roundrobin
>> > > >
>> > > > My producer.properties is:
>> > > >
>> > > > metadata.broker.list=[string of servers]
>> > > > compression.codec=gzip
>> > > > producer.type=async
>> > > > message.send.max.retries=2
>> > > >
>> > > > My understanding from the documentation is that this *should* work,
>> but
>> > > it
>> > > > is not.  All my data is going into partition 1 on the target cluster
>> > for
>> > > > all topics.
>> > > >
>> > > > Can someone help me understand what I'm missing here?
>> > > >
>> > > > Thank you for your time!
>> > > > Chris
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message