kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Thomas.K...@t-systems.com>
Subject WG: Exactly Once, Producer configuration
Date Mon, 14 May 2018 16:13:53 GMT
Hi,

the following article describes how to configure Apache Kafka system to enforce exactly once
approach:

https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

One recommendation states that parameter transactional.id should be set in producer.properties
file which usually exists only once per broker.

Now I have the following questions. Our system consists of 4 modules which interact in a sequential
way by exchanging messages using Kafka topics


A --> B --> C --> D

I.e. modules A, B and C act as message producers and modules B, C and D act as message consumers
(and all modules do some application specific useful processing).

1)      Do we need separate producer/consumer configurations for these modules and if so where
and how do we have to configure them (as producer configuration files are related to the number
of brokers and not to the number of modules)?
2)      Does transcational.id "only" cover technical aspects of Apache Kafka or does it cover
application aspects as well (i.e. is my assumption which led to question 1 correct)?
3)      In case it is sufficient to configure transactional.id parameter in producer.properties
files and we have more than one broker (and with that more than one producer.properties file).
Do the values of the parameter transactional.id have to be equal or do there have to be different
(pairwise disjoint) values?

I would be glad to get answers to these questions.

Kind regards,
Thomas Kass






Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message