kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Dead kafka consumers claim partitions, new ones can't claim any
Date Wed, 11 Jan 2012 21:30:42 GMT
Evan,

What you are describing seems to be this bug in zookeeper 3.3.3 -
https://issues.apache.org/jira/browse/ZOOKEEPER-1208

However, to confirm if thats the case, do you mind uploading the
entire log for the consumer ?

Thanks,
Neha

On Wed, Jan 11, 2012 at 11:33 AM, Evan Chan <ev@ooyala.com> wrote:
> Oh, here are the consumer logs when it can't claim a partition:
>
>
> 2012-01-11 02:23:04 ZookeeperConsumerConnector [WARN] No broker partions
> consumed by consumer thread
> whatever_ip-10-116-81-39.ec2.internal-1326248583706-0
> for t
> opic player_logs
> 2012-01-11 02:23:04 ZookeeperConsumerConnector [INFO] Consumer
> whatever_ip-10-116-81-39.ec2.internal-1326248583706 selected partitions :
>
>
> On Wed, Jan 11, 2012 at 11:32 AM, Evan Chan <ev@ooyala.com> wrote:
>
>> Hi,
>>
>> We are using Kafka 0.6 and testing it on EC2.  We have an issue where some
>> processes running the ZK/High level consumer (Scala consumer) get killed
>> before they have a chance to call ConsumerConnector.shutdown().
>> It seems like they leave nodes hanging around in ZK.
>> If we restart the process, the consumer in the new process will error out
>> because it cannot claim any partitions.
>>
>> The only way I know of getting around this is to use a ZK client and
>> manually delete nodes.
>>
>> Is there any way for the high level consumer nodes in ZK to be made
>> ephemeral so that if a process gets killed, the state won't last forever
>> and cause subsequent nodes to not be able to claim partitions?
>> Any chance this has been fixed in 0.7?
>>
>> thanks,
>> Evan
>>
>> --
>> --
>> *Evan Chan*
>> Senior Software Engineer |
>> ev@ooyala.com | (650) 996-4600
>> www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>

Mime
View raw message