kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tao li <tust.05102...@gmail.com>
Subject [Kafka-users] Differences MessageIn between two kafka clusters replicated by mirror maker
Date Tue, 06 Jan 2015 08:16:54 GMT
Hi,

We are using mirrormaker to replicate data between two kafka clusters in
difference datacenter, a fiber connected them, 1G bandwidth, ~35ms
lantency.

we found that the MessageIn in JMX beans with the broker in destination
cluster greater than broker in source cluster.

Our configuration like this:
destination & source cluster both deployed kafka 0.7.0
source clusters has 6 brokers with 32 partitions each,target clusters has
12 brokers with 32 partitions each and 6 mirrormakers doing
replication,replicate only one topic.

mirrormaker configuration like this:

--num.streams 128
--num.producers 16

=================================

consumer.properties:

shallowiterator.enable=true

socket.buffersize=31457280

fetch.size=1048576

=================================

producer.properties:

queue.enqueueTimeout.ms <http://queue.enqueuetimeout.ms/>=-1

# the number of messages batched at the producer

batch.size=4000

#default 5000

connect.timeout.ms=10000

#default 30000

socket.timeout.ms=300000

==================================

test result:

MessageIn: source cluster => 252,560,169, target cluster => 253,128,669

another problem, sometimes one or more broker in source clusters has not be
consumered during the mirrormakers replicate other brokers well.

Does anyone have an idea to resolve them.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message