kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik van Oosten <e.vanoos...@grons.nl.INVALID>
Subject Re: How to produce and consume events in 2 DCs?
Date Wed, 22 Oct 2014 07:21:08 GMT
Hi Steven,

That doesn't work. In your proposal mirrormaker in once DC would copy 
messages from topic A to the other DC in topic A. However, in the other 
DC there is a mirrormaker which does the same, creating a loop. Messages 
will be duplicated, triplicated, etc in a never ending loop.

Mirroring to another topic would work (mirrormaker doesn't support 
that), and so would mirroring to another cluster. Neha's proposal would 
work also but I assume its a lot more work for the Kafka internals and 
therefor IMHO wouldn't meet the kiss principle.

Kind regards,
     Erik.


Steven Wu schreef op 22-10-14 om 01:48:
> I think it doesn't have to be two more clusters. can be just two more
> topics. MirrorMaker can copy from source topics in both regions into one
> aggregate topic.
>
> On Tue, Oct 21, 2014 at 1:54 AM, Erik van oosten <
> e.vanoosten@grons.nl.invalid> wrote:
>
>> Thanks Neha,
>>
>> Unfortunately, the maintenance overhead of 2 more clusters is not
>> acceptable to us.
>>
>> Would you accept a pull request on mirror maker that would rename topics
>> on the fly?
>>
>> For example by accepting the parameter rename:
>>     —rename src1/dest1,src2/dest2
>> or, extended with RE support:
>>     —rename old_(.*)/new_\1
>>
>> Kind regards,
>>      Erik.
>>
>>
>> Op 20 okt. 2014, om 16:43 heeft Neha Narkhede <neha.narkhede@gmail.com>
>> het volgende geschreven:
>>
>>> Another way to set up this kind of mirroring is by deploying 2 clusters
>> in
>>> each DC - a local Kafka cluster and an aggregate Kafka cluster. The
>> mirror
>>> maker copies data from both the DC's local clusters into the aggregate
>>> clusters. So if you want access to a topic with data from both DC's, you
>>> subscribe to the aggregate cluster.
>>>
>>> Thanks,
>>> Neha
>>>
>>> On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten <
>>> e.vanoosten@grons.nl.invalid> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have 2 data centers that produce events. Each DC has to process
>> events
>>>> from both DCs.
>>>>
>>>> I had the following in mind:
>>>>
>>>>                    DC 1 | DC 2
>>>>         events          |            events
>>>>        +  +  +          |           +  +  +
>>>>        |  |  |          |           |  |  |
>>>>        v  v  v          |           v  v  v
>>>> +----------------+     |     +----------------+
>>>> | Receiver topic |     |     | Receiver topic |
>>>> +----------------+           +----------------+
>>>>          |  |       mirroring      |    |
>>>>          |  |   +------------------+    |
>>>>          |  |   |                       |
>>>>          |  +--------------------+      |
>>>>          v      v                v      v
>>>> +----------------+     |     +----------------+
>>>> | Consumer topic |     |     | Consumer topic |
>>>> +----------------+     |     +----------------+
>>>>        +  +  +          |           +  +  +
>>>>        |  |  |          |           |  |  |
>>>>        v  v  v          |           v  v  v
>>>>       consumers         |          consumers
>>>>
>>>>
>>>> As each DC has a single Kafka cluster, on each DC the receiver topic and
>>>> consumer topic needs to be on the same cluster.
>>>> Unfortunately, mirror maker does not seem to support mirroring to a
>> topic
>>>> with another name.
>>>>
>>>> Is there another tool we could use?
>>>> Or, is there another approach for producing and consuming from 2 DCs?
>>>>
>>>> Kind regards,
>>>>     Erik.
>>>>
>>>> —
>>>> Erik van Oosten
>>>> http://www.day-to-day-stuff.blogspot.nl/
>>>>
>>>>
>>


-- 
Erik van Oosten
http://www.day-to-day-stuff.blogspot.com/


Mime
View raw message