kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik van oosten <e.vanoos...@grons.nl.INVALID>
Subject Re: How to produce and consume events in 2 DCs?
Date Tue, 21 Oct 2014 08:54:01 GMT
Thanks Neha,

Unfortunately, the maintenance overhead of 2 more clusters is not acceptable to us.

Would you accept a pull request on mirror maker that would rename topics on the fly?

For example by accepting the parameter rename:
   —rename src1/dest1,src2/dest2
or, extended with RE support:
   —rename old_(.*)/new_\1

Kind regards,
    Erik.


Op 20 okt. 2014, om 16:43 heeft Neha Narkhede <neha.narkhede@gmail.com> het volgende
geschreven:

> Another way to set up this kind of mirroring is by deploying 2 clusters in
> each DC - a local Kafka cluster and an aggregate Kafka cluster. The mirror
> maker copies data from both the DC's local clusters into the aggregate
> clusters. So if you want access to a topic with data from both DC's, you
> subscribe to the aggregate cluster.
> 
> Thanks,
> Neha
> 
> On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten <
> e.vanoosten@grons.nl.invalid> wrote:
> 
>> Hi,
>> 
>> We have 2 data centers that produce events. Each DC has to process events
>> from both DCs.
>> 
>> I had the following in mind:
>> 
>>                   DC 1 | DC 2
>>        events          |            events
>>       +  +  +          |           +  +  +
>>       |  |  |          |           |  |  |
>>       v  v  v          |           v  v  v
>> +----------------+     |     +----------------+
>> | Receiver topic |     |     | Receiver topic |
>> +----------------+           +----------------+
>>         |  |       mirroring      |    |
>>         |  |   +------------------+    |
>>         |  |   |                       |
>>         |  +--------------------+      |
>>         v      v                v      v
>> +----------------+     |     +----------------+
>> | Consumer topic |     |     | Consumer topic |
>> +----------------+     |     +----------------+
>>       +  +  +          |           +  +  +
>>       |  |  |          |           |  |  |
>>       v  v  v          |           v  v  v
>>      consumers         |          consumers
>> 
>> 
>> As each DC has a single Kafka cluster, on each DC the receiver topic and
>> consumer topic needs to be on the same cluster.
>> Unfortunately, mirror maker does not seem to support mirroring to a topic
>> with another name.
>> 
>> Is there another tool we could use?
>> Or, is there another approach for producing and consuming from 2 DCs?
>> 
>> Kind regards,
>>    Erik.
>> 
>> —
>> Erik van Oosten
>> http://www.day-to-day-stuff.blogspot.nl/
>> 
>> 


Mime
View raw message