kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vito Sabella <vsabe...@gmail.com>
Subject Error Handling and Acknowledgements in Kafka Consumers‏
Date Thu, 13 Sep 2012 21:10:09 GMT
** Resending due to outlook.com corrupting the original message. Sorry :) **


I'm looking to use kafka in a pub-sub model where a consumer reads
from Kafka and does some processing on the message.  How would you
recommend a commit to Zookeeper / setting the last message consumed
location if processing one of the messages in the pipe is more
unreliable than the others.

Let's say I read a batch of 10 messages (1-10) and I successfully
process messages 1-8 and 10 quickly, but message #9 is taking an
inordinately long time to process. I don't want to write the message
as consumed against Zookeeper but I also don't want to block forward
progress of the pipeline for that topic/partition.

In the edge case, let's say processing all messages was successful but
processing message #9 totally failed/timed out, but a reattempt at
processing it would not result in failure (web-service call, strange
network condition). In the timeout/failure case would the suggestion
be to re-queue the message?

Does anyone have any recommendations for the above scenarios?


View raw message