kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject Re: High level consumer replaying the messages
Date Thu, 10 Apr 2014 04:47:35 GMT
I see this a lot in the consumer logs
  [kafka.utils.ZkUtils$] conflict in 
/consumers/group1/owners/testtopic/2 data: 
group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: 
group1_ip-10-168-47-114-1397034341089-3861c6cc-1

I am not seeing any "expired" in my consumer logs

Thanks
Arjun Narasimha Kota

On Thursday 10 April 2014 09:13 AM, Jun Rao wrote:
> Do you see many rebalances in the consumer log? If so, see
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> ?
>
> Thanks,
>
> Jun
>
>
> On Wed, Apr 9, 2014 at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:
>
>> Hi ,
>>
>> I have set up kafka 0.8 in 3 servers. I have pushed some data into these
>> servers. The number of partitions i use is 12, with a replication factor of
>> 2.
>> We use high level consumer to consume messages. we set auto commit to
>> false, and we do commits almost after each and every successful message
>> consumption.
>> The problem i am facing now is, If i restart my consumers, then the
>> messages are getting replayed.
>>
>> I checked the
>>
>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
>> --zkconnect zkhost:zkport --topic testtopic
>>
>> I see a high lag in some of the partitions and some partitions have none
>> as the owner.
>> To verify consumer rebalance i used the following command
>>
>> bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group group1
>> --zookeeper.connect zkhost:zkport
>>
>> but i see errors
>>
>> [2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
>> (kafka.tools.VerifyConsumerRebalance$)
>>
>>
>> I have seen this ( http://mail-archives.apache.org/mod_mbox/kafka-users/
>> 201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM
>> j_eSy0YQUCoEMoxg@mail.gmail.com%3E) in the mail archives and i checked my
>> consumer log for those patterns but couldn't find any in my logs.
>>
>> Can some one help me with this.
>>
>> Thanks
>> Arjun Narasimha kota
>>


Mime
View raw message