kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: [Kafka-users] Differences MessageIn between two kafka clusters replicated by mirror maker
Date Wed, 07 Jan 2015 06:17:07 GMT
Not sure exactly what's happening there. In any case, 0.7 is really old.
You probably want to upgrade to the latest 0.8 release.

Thanks,

Jun

On Tue, Jan 6, 2015 at 12:16 AM, tao li <tust.05102405@gmail.com> wrote:

> Hi,
>
> We are using mirrormaker to replicate data between two kafka clusters in
> difference datacenter, a fiber connected them, 1G bandwidth, ~35ms
> lantency.
>
> we found that the MessageIn in JMX beans with the broker in destination
> cluster greater than broker in source cluster.
>
> Our configuration like this:
> destination & source cluster both deployed kafka 0.7.0
> source clusters has 6 brokers with 32 partitions each,target clusters has
> 12 brokers with 32 partitions each and 6 mirrormakers doing
> replication,replicate only one topic.
>
> mirrormaker configuration like this:
>
> --num.streams 128
> --num.producers 16
>
> =================================
>
> consumer.properties:
>
> shallowiterator.enable=true
>
> socket.buffersize=31457280
>
> fetch.size=1048576
>
> =================================
>
> producer.properties:
>
> queue.enqueueTimeout.ms <http://queue.enqueuetimeout.ms/>=-1
>
> # the number of messages batched at the producer
>
> batch.size=4000
>
> #default 5000
>
> connect.timeout.ms=10000
>
> #default 30000
>
> socket.timeout.ms=300000
>
> ==================================
>
> test result:
>
> MessageIn: source cluster => 252,560,169, target cluster => 253,128,669
>
> another problem, sometimes one or more broker in source clusters has not be
> consumered during the mirrormakers replicate other brokers well.
>
> Does anyone have an idea to resolve them.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message