kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zach Cox <zcox...@gmail.com>
Subject Transaction commit markers and topic partition end offsets
Date Wed, 06 Mar 2019 17:08:34 GMT
Hi - we've observed that after committing a transaction, commit marker
records are appended to all topic partitions written to in the transaction,
these commit markers are assigned offsets in the partition, and
KafkaConsumer.endOffsets will return the offsets of commit markers if they
are the last records appended to the partition, although the consumer will
never consume records with those commit marker offsets.

This can lead to problems, for example on startup a system gets current end
offsets for a topic's partitions and tracks the current offsets a consumer
has consumed, it will never be "caught up" to the last offsets since they
are the commit marker offsets, until newer records are written to those
partitions. A workaround to this specific problem is to assume lastOffset =
endOffset - 2, but at best this is surprising until you learn about commit
markers.

How have others dealt with this? Are we missing anything important in the
above? Could the docs include more details about commit markers? Should
KafkaConsumer.endOffsets account for commit markers somehow?

Thanks,
Zach

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message