kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Sometimes get no message streams in consumer
Date Thu, 05 Sep 2013 03:40:53 GMT
What createMessageStreams() returns should match what's in the input. Is it
possible for getTopicMap() to return an empty map?

Thanks,

Jun


On Wed, Sep 4, 2013 at 1:43 PM, Ian Friedman <ian@flurry.com> wrote:

> Hey gang, I have a strange intermittent issue with our 0.7.2 consumers
> that is perpelxing me:
>
> It seems like, sometimes very infrequently, when restarting our Consumer
> fleet, some consumers threads just fail to start. The consumer connector
> itself starts fine, and seems to be involved in subsequent rebalances, but
> based on the logging I'm seeing, it looks like our one call to
> ConsumerConnector.createMessageStreams is returning an empty map.
>
> Is that something that should happen normally and should we be retrying in
> our calling code? Our consumer code looks like this:
>
> Map<String, List<KafkaStream<Message>>> topicMessageStreams =
>    fConsumerConnector.createMessageStreams( getTopicMap() );
>
> for( Map.Entry<String, List<KafkaStream<Message>>> entry :
> topicMessageStreams.entrySet() )
> {
> final String topicName = entry.getKey();
> log.info( "starting stream for topic: " + topicName );
> final KafkaStream<Message> stream = entry.getValue().get(0);
>
> ….
>
> Are we doing anything dumb here? In the case I described, we're never
> getting the "starting stream for topic: " message in our log output, so my
> only guess is that createMessageStreams is returning an empty map.
>
> Thanks in advance guys!
>
> --
> Ian Friedman
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message