kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Friedman <...@flurry.com>
Subject Sometimes get no message streams in consumer
Date Wed, 04 Sep 2013 20:43:38 GMT
Hey gang, I have a strange intermittent issue with our 0.7.2 consumers that is perpelxing me:

It seems like, sometimes very infrequently, when restarting our Consumer fleet, some consumers
threads just fail to start. The consumer connector itself starts fine, and seems to be involved
in subsequent rebalances, but based on the logging I'm seeing, it looks like our one call
to ConsumerConnector.createMessageStreams is returning an empty map.  

Is that something that should happen normally and should we be retrying in our calling code?
Our consumer code looks like this:

Map<String, List<KafkaStream<Message>>> topicMessageStreams =  
   fConsumerConnector.createMessageStreams( getTopicMap() );

for( Map.Entry<String, List<KafkaStream<Message>>> entry : topicMessageStreams.entrySet()
final String topicName = entry.getKey();  
log.info( "starting stream for topic: " + topicName );
final KafkaStream<Message> stream = entry.getValue().get(0);


Are we doing anything dumb here? In the case I described, we're never getting the "starting
stream for topic: " message in our log output, so my only guess is that createMessageStreams
is returning an empty map.  

Thanks in advance guys!

Ian Friedman

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message