kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiri Humpolicek <jiri.humpoli...@firma.seznam.cz>
Subject Does MirrorMaker ensures exactly-once?
Date Mon, 08 Jan 2018 11:36:21 GMT
Hi Everyone,

since kafka 0.11.x supports exactly-once semantics, I want to be sure,
that it is possible to achieve it across kafka clusters using MirrorMaker.

We have got two locations with "primary" cluster in each location and
for each location we have got one "aggregation" cluster which mirrors
data from all primary clusters.

Currently we deduplicate messages when we copying data from aggregation
kafka to HDFS by separete YARN application. But in aggregation kafka
duplicates remains. So I want to ensure that there are no duplicates and
loss data in kafka and not to use our deduplication yarn application
anymore.

If it is possible, how to configure MirrorMaker to achieve exactly-once
delivery across primary and aggregation clusters?


Thanks and have a nice day, Jiri Humpolicek



Je dobré vědět, že tento e-mail a přílohy jsou důvěrné. Pokud spolu jednáme o uzavření
obchodu, vyhrazujeme si právo naše jednání kdykoli ukončit. Pro fanoušky právní mluvy
- vylučujeme tím ustanovení občanského zákoníku o předsmluvní odpovědnosti. Pravidla
o tom, kdo u nás a jak vystupuje za společnost a kdo může co a jak podepsat naleznete
zde<https://onas.seznam.cz/cz/podpisovy-rad-cz.html>

You should know that this e-mail and its attachments are confidential. If we are negotiating
on the conclusion of a transaction, we reserve the right to terminate the negotiations at
any time. For fans of legalese—we hereby exclude the provisions of the Civil Code on pre-contractual
liability. The rules about who and how may act for the company and what are the signing procedures
can be found here<https://onas.seznam.cz/cz/signature-rules.html>.

Mime
View raw message