kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sites <Eric.Si...@threattrack.com>
Subject Re: Unpacking "strong durability and fault-tolerance guarantees"
Date Sat, 13 Jul 2013 02:57:06 GMT

In a high-level consumer "ConsumerConnector" you can setup the property
auto.commit.enable = false. Then you can call commitOffsets after
processing the message.

Calling commitOffsets after every message will slow processing down
dramatically. So setup a timer or other coordination mechanism on your
code to commitOffsets after every x number of messages.

Eric Sites

On 7/12/13 9:39 PM, "Florin Trofin" <ftrofin@adobe.com> wrote:

>Thanks Jay, this is helpful information. Controlling when offsets are
>committed is possible only with the SimpleConsumer, correct? I believe
>that the ConsumerConnector (the high level consumer) commits the offsets
>automatically. It would be nice to have a hook into it that I can
>override to let it know when I finished processing a message so it can
>commit an offset. Is there an improvement request submitted for this
>On Jul 7, 2013, at 2:35 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
>> Consumer:
>> The consumers position is controlled using a saved "offset" that marks
>> position in the topic/partition it is reading. This position is
>> periodically updated. If you update the saved offset before processing
>> messages you have the possibility of message loss if your consumer
>> before processing the messages. If you update the offset after
>> the messages you have the possibility of duplicate messages when your
>> consumer restarts as it will reprocess a few message it has already
>> You can control when the position is saved by calling commit().

View raw message