kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Schmitt <killas...@gmail.com>
Subject Strange mirroring problem with kafka 0.8.2.1
Date Thu, 01 Oct 2015 18:35:27 GMT
I’m trying to replicate data from 1 DC to a remote DC in AWS.

Have a node in DC1, running Kafka 0.8.2.1 under zookeeper.
- hostname = is11
- broker.id = 0
- port 6667

MirrorMaker in DC2, running Kafka 0.8.2.1
- hostname = zoo1001

consumer.config

auto.commit.enable=true
auto.commit.interval.ms=10000
auto.offset.reset=smallest
consumer.timeout.ms=-1
fetch.message.max.bytes=1048576
fetch.min.bytes=1
fetch.wait.max.ms=100
group.id=elasticsearch
queued.max.message.chunks=10
rebalance.backoff.ms=2000
rebalance.max.retries=4
refresh.leader.backoff.ms=200
socket.receive.buffer.bytes=65536
socket.timeout.ms=30000
zookeeper.connect=is11:2181
zookeeper.connection.timeout.ms=6000
zookeeper.session.timeout.ms=6000
zookeeper.sync.time.ms=2000
offsets.storage=kafka

producer.config

batch.num.messages=200
compression.codec=gzip
message.send.max.retries=10
metadata.broker.list=10.10.7.91:6667
partitioner.class=kafka.producer.DefaultPartitioner
producer.type=async
queue.buffering.max.messages=10000
queue.buffering.max.ms=5000
queue.enqueue.timeout.ms=-1
request.required.acks=-1
request.timeout.ms=10000
retry.backoff.ms=10000
send.buffer.bytes=102400
serializer.class=kafka.serializer.DefaultEncoder
topic.metadata.refresh.interval.ms=60000
client.id=elasticsearch-usw2
reconnect.backoff.ms=1000

Kafka Broker in DC2, running Kafka 0.8.2.1 under zookeeper

- hostname = zoo1002 (10.10.7.91)
- broker.id = 1
- port 6667

Mirror maker is consuming topics just fine, but when the producer thread starts up initially,
it never connects to zoo1002. The thread reverts to using is11 as the metadatabroker instead.
Am I missing some setting here? Or other troubleshooting ideas I may have missed?

If I start up a consumer thread manually and then a producer thread that targets a specific
topic. it is copied to zoo1002 w/o issue. It’s just when running under mirror maker that
there is a problem.

Thanks.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message