kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Shutdown/Ctrl-C and ConsumerRebalanceFailedException
Date Tue, 20 Mar 2012 16:21:27 GMT
Peter,

When you kill your consumer, it takes sometime (by default 6 secs) for ZK
server to release the ephemeral nodes hold by the consumer. If you restart
your consumer quickly, the consumer may not be able to acquire the
necessary zk nodes and therefore can fail to rebalance. One way is to
implement a shutdown hook in your consumer application and make sure
consumer connector is shut down when the consumer is killed.

Thanks,

Jun

On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activemq@gmail.com>wrote:

> When I shutdown my consumer with crtl-c and tries to restart it quickly
> afterwards, I usually get ConsumerRebalanceFailedException (see below). The
> application then seems to hang.. or at least I'm sure if it is running any
> more.. If this exception is thrown, will the consumer the intelligently
> wait for the rebalancing to complete? and then resume consumption?
>
> I found a page
> https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat
> describes something about Consumer Co-ordinator.. according to this
> the consumer
> group remains in this state until the next rebalancing attempt is
> triggered. But when is it triggered?
>
> Could a shutdown hock with a consumer.commitOffsets help?
> Does the consumer.shutdown implicit commitOffsets?
>
>
> Exception in thread "main" kafka.common.ConsumerRebalanceFailedException:
> contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't
> rebalance after 4 retries
>        at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467)
>        at
>
> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204)
>        at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75)
>        at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89)
>        at
>
> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75)
>        at
>
> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111)
>
>
> Brgds,
> Peter Thygesen
>
> BTW: Great work, very interesting project.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message