kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Can Mirroring Preserve Every Topic's Partition?
Date Wed, 26 Nov 2014 18:33:56 GMT
Hello Alex,

This can be done by doing some tweaks in the MM code (with the 0.8.2 new
producer).

1. Set-up your MM to have the total # of producers equal to the #. of
partitions in source / target cluster.

2. When the consumer of the MM gets a message, put the message to the
producer's queue based on its partition id; i.e. if the partition id is n,
put to n's producer queue.

3. When producer sends the data, specify the partition id; so each producer
will only send to a single partition.

Guozhang


On Tue, Nov 25, 2014 at 8:19 PM, Alex Melville <amelville@g.hmc.edu> wrote:

> Howdy friends,
>
>
> I'd like to mirror the topics on several clusters to a central cluster, and
> I'm looking at using the default Mirrormaker to do so. I've already done
> some basic testing on the Mirrormaker found here:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
>
> and managed to successfully copy a topic's partitions on a source cluster
> to a topic on a target cluster. So I'm able to mirror correctly. However
> for my particular use case I need to ensure that when I copy a topic's
> partitions from source cluster to target cluster, a partition created on
> the target cluster contains data in the exact same order as the data on the
> corresponding partition on the source cluster.
>
> I'm thinking of writing a Simple Consumer so I can manually compare the
> events in a source cluster's partition with the corresponding partition on
> the target cluster, but I'm not 100% sure if I'll be able to verify my
> guarantee if I do it this way. Can anyone here verify that partitions
> copied over to the target cluster by the default Mirrormaker are an exact
> copy of those on the source cluster?
>
>
> Thanks in advance,
>
> Alex Melville
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message