kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xander Uiterlinden <uiter...@gmail.com>
Subject Kafka streams exactly_once auto commit timeout transaction issue
Date Wed, 06 Feb 2019 14:14:02 GMT
Hi,

I'm trying to get a fairly simple example of using Kafka Streams with
exactly once processing to work. I defined a setup where messages are being
read from an input topic and two streams transform and output the result to
their own output topic.
In normal conditions this works fine, i.e. when publishing a message to the
input topic, I get transformed messages in both of the output topics.
Next I enabled exactly once processing by setting "processing.guarantee" to
"exactly_once". To test this I'm deliberately throwing an exception when
transforming the message in one of the stream processors. At a first glance
the result is as expected, as neither of the output topics contain the
transformed message and the application stops processing.
However, when processing a message takes longer than the commit.interval.ms
(which defaults to 100 when using exactly_once), then the transactional
guarantee does not appear to be there and I get an output message in only
one of the output topics (the one I for which I did not deliberately throw
an exception while processing). I tested this by putting a Thread.sleep()
before throwing the Exception.
Can someone explain the relationship between this commit.interval.ms and
exactly_once processing. I'd think it's rather strange that when processing
takes longer than the commit.interval.ms you lose the atomicity of the
transaction.
I'm using Kafka 2.12-1.1.0 by the way.

Kind regards,

Xander

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message