kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Trinity Xia <trini...@scribd.com>
Subject Kafka Streams - offset number affected by "exactly once processing guarantee"
Date Thu, 20 Jun 2019 21:42:38 GMT
Hi,

We are using Kafka Streams 2.2.1 and Kafka 2.2.0, and we noticed that the
end offset number is larger than the numbers of events sent to a topic if
we set *processing guarantee* as *exactly once* in a Kafka Streams app.

For example, if we send 2615 events to an empty topic, we expect the end of
the topic to be offset 2616. However,  if we set *processing
guarantee* as *exactly
once,* the end of the topic is offset 2619 instead.

Key: 5-****5 offset: 0
Key: 5-****-5 offset: 1
Key: 5-****-5 offset: 2
Key: 5-****-5 offset: 3
Key: 5-****-5 offset: 4
Key: 25-****-f offset: 6
Key: 25-****-f offset: 7
Key: 25-****-f offset: 8
.....

% Reached end of topic tracking_events[0] at offset 2619: exiting

In addition, there are "missing offsets" in the topic. There seems to be
nothing inside message offset 5, but it still counts as a message...I used
Kafkacat to print out that message, and I have specified  -Z (Print NULL
messages and keys as "NULL"(instead of empty)), but it didn't print out
anything either...

Has anyone encountered the same issue before?  Are there any other methods
to calculate the number of messages inside a topic efficiently?  Thanks for
the help :)

Trinity

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message