kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Indu V <in...@suntecgroup.com>
Subject Finding consumer Lag
Date Mon, 08 Jun 2020 13:17:20 GMT
Hi Team,

I am facing an issue in finding the lag (unconsumed record count) from Kafka against a topic.
I have another process that has to be run after all records are consumed.

Each topic has given a single partition with the replication factor as 1.

Lag is computed by taking the difference of published offset and consumed offset.

In a transactional scenario, the published offset which is the partition end offset includes
control messages. In my understanding, this is internally added by Kafka to handle the transaction
commit and rollback.

I got count mismatches in a transaction rollback scenario. For a single unconsumed record,
it is showing lag as 3.

So is there any way I can exclude the control messages from the lag.

Indu V

This electronic mail (including any attachment thereto) may be confidential and privileged
and is intended only for the individual or entity named above. Any unauthorized use, printing,
copying, disclosure or dissemination of this communication may be subject to legal restriction
or sanction. Accordingly, if you are not the intended recipient, please notify the sender
by replying to this email immediately and delete this email (and any attachment thereto) from
your computer system...Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message