kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Thygesen <pt.activ...@gmail.com>
Subject Re: Shutdown/Ctrl-C and ConsumerRebalanceFailedException
Date Wed, 21 Mar 2012 12:08:56 GMT
I'm running kafka-0.7 on a hbase cluster where the zookeepers are from
Cloudera's CHD3 Update 2 = zookeeper-3.3.3

thank you for you time and support :)

I'll try and add a shutdown hook and close the consumer more "nicely".

/Peter

Den 20. mar. 2012 16.49 skrev Neha Narkhede <neha.narkhede@gmail.com>:

> Peter,
>
> >> If this exception is thrown, will the consumer the intelligently wait
> for the rebalancing to complete? and then resume consumption?
>
> If this exception is thrown, it means that the consumer has failed the
> current rebalancing attempt and will try only when one of the
> following happens -
>
> 1. New partitions are added to the topic it is consuming
> 2. Existing partitions become unavailable
> 3. New consumer instances are brought up for the consumer group it belongs
> to
> 4. Existing consumer instances die for the consumer group it belongs to
>
> Until that, the consumer is not fully functional. So, this particular
> exception should be monitored and the consumer instance should be
> restarted.
>
> Having said that, it is pretty rare for the consumer to run out of
> rebalancing attempts. One of the common causes is using zookeeper
> 3.3.3 which causes older ephemeral nodes to be retained.
> Which version of Kafka are you using ?
> Would you mind attaching the entire log for the consumer. It will help
> us debug the cause of this exception and see if it is an actual bug.
>
> Thanks,
> Neha
>
>
> On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activemq@gmail.com>
> wrote:
> > When I shutdown my consumer with crtl-c and tries to restart it quickly
> > afterwards, I usually get ConsumerRebalanceFailedException (see below).
> The
> > application then seems to hang.. or at least I'm sure if it is running
> any
> > more.. If this exception is thrown, will the consumer the intelligently
> > wait for the rebalancing to complete? and then resume consumption?
> >
> > I found a page
> https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat
> > describes something about Consumer Co-ordinator.. according to this
> > the consumer
> > group remains in this state until the next rebalancing attempt is
> > triggered. But when is it triggered?
> >
> > Could a shutdown hock with a consumer.commitOffsets help?
> > Does the consumer.shutdown implicit commitOffsets?
> >
> >
> > Exception in thread "main" kafka.common.ConsumerRebalanceFailedException:
> > contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't
> > rebalance after 4 retries
> >        at
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467)
> >        at
> >
> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204)
> >        at
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75)
> >        at
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89)
> >        at
> >
> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75)
> >        at
> >
> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111)
> >
> >
> > Brgds,
> > Peter Thygesen
> >
> > BTW: Great work, very interesting project.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message