kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Shutdown/Ctrl-C and ConsumerRebalanceFailedException
Date Tue, 20 Mar 2012 15:49:08 GMT
Peter,

>> If this exception is thrown, will the consumer the intelligently wait for the rebalancing
to complete? and then resume consumption?

If this exception is thrown, it means that the consumer has failed the
current rebalancing attempt and will try only when one of the
following happens -

1. New partitions are added to the topic it is consuming
2. Existing partitions become unavailable
3. New consumer instances are brought up for the consumer group it belongs to
4. Existing consumer instances die for the consumer group it belongs to

Until that, the consumer is not fully functional. So, this particular
exception should be monitored and the consumer instance should be
restarted.

Having said that, it is pretty rare for the consumer to run out of
rebalancing attempts. One of the common causes is using zookeeper
3.3.3 which causes older ephemeral nodes to be retained.
Which version of Kafka are you using ?
Would you mind attaching the entire log for the consumer. It will help
us debug the cause of this exception and see if it is an actual bug.

Thanks,
Neha


On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activemq@gmail.com> wrote:
> When I shutdown my consumer with crtl-c and tries to restart it quickly
> afterwards, I usually get ConsumerRebalanceFailedException (see below). The
> application then seems to hang.. or at least I'm sure if it is running any
> more.. If this exception is thrown, will the consumer the intelligently
> wait for the rebalancing to complete? and then resume consumption?
>
> I found a page https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat
> describes something about Consumer Co-ordinator.. according to this
> the consumer
> group remains in this state until the next rebalancing attempt is
> triggered. But when is it triggered?
>
> Could a shutdown hock with a consumer.commitOffsets help?
> Does the consumer.shutdown implicit commitOffsets?
>
>
> Exception in thread "main" kafka.common.ConsumerRebalanceFailedException:
> contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't
> rebalance after 4 retries
>        at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467)
>        at
> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204)
>        at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75)
>        at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89)
>        at
> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75)
>        at
> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111)
>
>
> Brgds,
> Peter Thygesen
>
> BTW: Great work, very interesting project.

Mime
View raw message