kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Burroughs <chris.burrou...@gmail.com>
Subject Re: Shutdown/Ctrl-C and ConsumerRebalanceFailedException
Date Mon, 26 Mar 2012 02:01:29 GMT
If the consumer is cleanly shutdown (Control-C is kill, not kill -9,
right?), shoudn't we remove the ephemeral ZNodes in our shutdown
handler?  I thought this was already the case, at least for the broker.

On 03/20/2012 12:21 PM, Jun Rao wrote:
> Peter,
> 
> When you kill your consumer, it takes sometime (by default 6 secs) for ZK
> server to release the ephemeral nodes hold by the consumer. If you restart
> your consumer quickly, the consumer may not be able to acquire the
> necessary zk nodes and therefore can fail to rebalance. One way is to
> implement a shutdown hook in your consumer application and make sure
> consumer connector is shut down when the consumer is killed.
> 
> Thanks,
> 
> Jun
> 
> On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activemq@gmail.com>wrote:
> 
>> When I shutdown my consumer with crtl-c and tries to restart it quickly
>> afterwards, I usually get ConsumerRebalanceFailedException (see below). The
>> application then seems to hang.. or at least I'm sure if it is running any
>> more.. If this exception is thrown, will the consumer the intelligently
>> wait for the rebalancing to complete? and then resume consumption?
>>
>> I found a page
>> https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat
>> describes something about Consumer Co-ordinator.. according to this
>> the consumer
>> group remains in this state until the next rebalancing attempt is
>> triggered. But when is it triggered?
>>
>> Could a shutdown hock with a consumer.commitOffsets help?
>> Does the consumer.shutdown implicit commitOffsets?
>>
>>
>> Exception in thread "main" kafka.common.ConsumerRebalanceFailedException:
>> contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't
>> rebalance after 4 retries
>>        at
>>
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467)
>>        at
>>
>> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204)
>>        at
>>
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75)
>>        at
>>
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89)
>>        at
>>
>> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75)
>>        at
>>
>> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111)
>>
>>
>> Brgds,
>> Peter Thygesen
>>
>> BTW: Great work, very interesting project.
>>
> 


Mime
View raw message