kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ross <dyr...@klout.com>
Subject [0.6] Consumer API Question - What if a ConsumerIterator has an error?
Date Thu, 22 Dec 2011 09:40:46 GMT

I have come across a potential issue with the Consumer API in Kafka 0.6.
After some tests and reading through the code, I believe the following
scenario could result in an unhappy situation:

   1. Create 5 streams listening to a topic
   2. One of the streams times out (or has some other error) and it's
   iterator is now in mode FAILED
   3. (I think) Messages will now come into all five streams, but one of
   them won't yield any new messages because of (2)

The only solution I can think of is if an error is detected in any of the
streams, all of them should be stopped and a new set of streams should be
made. This seems unfavorable and would require messy concurrent programming.

So is this an actual problem? Is there a better workaround? Have I
misinterpreted something?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message