kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From S Ahmed <sahmed1...@gmail.com>
Subject RE: storing last processed offset, recovery of failed message processing etc.
Date Mon, 09 Dec 2013 20:28:53 GMT
I was hoping people could comment on how they handle the following
scenerios:

1. Storing the last successfully processed messageId/Offset.  Are people
using mysql, redis, etc.?  What are the tradeoffs here?

2. How do you handle recovering from an error while processesing a given
event?

There are various scenerioes for #2, like:
1. Do you mark the start of processing a message somewhere, and then update
the status to complete and THEN update the last messaged processed for #1?
2. Do you only mark the status as complete, and not the start of processing
it?  I guess this depends of there are intermediate steps and processing
the entire message again would result in some duplicated work right?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message