kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Shutdown/Ctrl-C and ConsumerRebalanceFailedException
Date Mon, 26 Mar 2012 02:08:04 GMT
Peter,

If you are using zk namespaces on zk 3.3.3, you will most certainly
hit the ephemeral node issue.

Chris,

If the consumer is shutdown, cleanly or not, zookeeper deletes the
ephemeral nodes from its database.
(unless you are using zk 3.3.3)

Thanks,
Neha

On Sun, Mar 25, 2012 at 7:01 PM, Chris Burroughs
<chris.burroughs@gmail.com> wrote:
> If the consumer is cleanly shutdown (Control-C is kill, not kill -9,
> right?), shoudn't we remove the ephemeral ZNodes in our shutdown
> handler?  I thought this was already the case, at least for the broker.
>
> On 03/20/2012 12:21 PM, Jun Rao wrote:
>> Peter,
>>
>> When you kill your consumer, it takes sometime (by default 6 secs) for ZK
>> server to release the ephemeral nodes hold by the consumer. If you restart
>> your consumer quickly, the consumer may not be able to acquire the
>> necessary zk nodes and therefore can fail to rebalance. One way is to
>> implement a shutdown hook in your consumer application and make sure
>> consumer connector is shut down when the consumer is killed.
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activemq@gmail.com>wrote:
>>
>>> When I shutdown my consumer with crtl-c and tries to restart it quickly
>>> afterwards, I usually get ConsumerRebalanceFailedException (see below). The
>>> application then seems to hang.. or at least I'm sure if it is running any
>>> more.. If this exception is thrown, will the consumer the intelligently
>>> wait for the rebalancing to complete? and then resume consumption?
>>>
>>> I found a page
>>> https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat
>>> describes something about Consumer Co-ordinator.. according to this
>>> the consumer
>>> group remains in this state until the next rebalancing attempt is
>>> triggered. But when is it triggered?
>>>
>>> Could a shutdown hock with a consumer.commitOffsets help?
>>> Does the consumer.shutdown implicit commitOffsets?
>>>
>>>
>>> Exception in thread "main" kafka.common.ConsumerRebalanceFailedException:
>>> contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't
>>> rebalance after 4 retries
>>>        at
>>>
>>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467)
>>>        at
>>>
>>> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204)
>>>        at
>>>
>>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75)
>>>        at
>>>
>>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89)
>>>        at
>>>
>>> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75)
>>>        at
>>>
>>> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111)
>>>
>>>
>>> Brgds,
>>> Peter Thygesen
>>>
>>> BTW: Great work, very interesting project.
>>>
>>
>

Mime
View raw message