kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ross Black <ross.w.bl...@gmail.com>
Subject Exactly-once semantics with compression
Date Fri, 01 Jun 2012 03:05:24 GMT

Using SimpleConsumer, I get the offset of a message (from MessageAndOffset)
and persist it with my consumer data to get exactly-once semantics for
consumer state (as described in the kafka design docs).  If the consumer
fails then it is simply a matter of starting replay of messages from the
persisted index.

When using compression, the offset from MessageAndOffset appears to be the
offset of the compressed batch.  e.g. For a batch of 10 messages, the
offset returned for messages 1-9 is the start of the *current* batch, and
the offset for message 10 is the start of the *next* batch.

How can I get the exactly-once semantics for consumer state?
Is there a way that I can get a batch of messages from SimpleConsumer?
(otherwise I have to reconstruct a batch by watching for a change in the
offset between messages)


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message