kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject Re: High level consumer replaying the messages
Date Thu, 10 Apr 2014 06:57:39 GMT
My consumer configuration has auto.offset.reset=smallest will this be 
causing the problem for replaying the messages?
I got to know why some partitions have no owners. It was my mistake i 
have 12 partitions but i started only 9 consumers. So when i check the 
Consumer Offset checker tool 3 partitions were showing "none" as the owner.

If some one can please help why we get this
"
[kafka.utils.ZkUtils$] conflict in /consumers/group1/owners/testtopic/2 
data: group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: 
group1_ip-10-168-47-114-1397034341089-3861c6cc-1
"
and the replay of messages i would be grateful

Thanks
Arjun Narasimha Kota


On Thursday 10 April 2014 10:17 AM, Arjun wrote:
> I see this a lot in the consumer logs
>  [kafka.utils.ZkUtils$] conflict in 
> /consumers/group1/owners/testtopic/2 data: 
> group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: 
> group1_ip-10-168-47-114-1397034341089-3861c6cc-1
>
> I am not seeing any "expired" in my consumer logs
>
> Thanks
> Arjun Narasimha Kota
>
> On Thursday 10 April 2014 09:13 AM, Jun Rao wrote:
>> Do you see many rebalances in the consumer log? If so, see
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog

>>
>> ?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Wed, Apr 9, 2014 at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:
>>
>>> Hi ,
>>>
>>> I have set up kafka 0.8 in 3 servers. I have pushed some data into 
>>> these
>>> servers. The number of partitions i use is 12, with a replication 
>>> factor of
>>> 2.
>>> We use high level consumer to consume messages. we set auto commit to
>>> false, and we do commits almost after each and every successful message
>>> consumption.
>>> The problem i am facing now is, If i restart my consumers, then the
>>> messages are getting replayed.
>>>
>>> I checked the
>>>
>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
>>> --zkconnect zkhost:zkport --topic testtopic
>>>
>>> I see a high lag in some of the partitions and some partitions have 
>>> none
>>> as the owner.
>>> To verify consumer rebalance i used the following command
>>>
>>> bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group 
>>> group1
>>> --zookeeper.connect zkhost:zkport
>>>
>>> but i see errors
>>>
>>> [2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>> [2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
>>> (kafka.tools.VerifyConsumerRebalance$)
>>>
>>>
>>> I have seen this ( 
>>> http://mail-archives.apache.org/mod_mbox/kafka-users/
>>> 201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM
>>> j_eSy0YQUCoEMoxg@mail.gmail.com%3E) in the mail archives and i 
>>> checked my
>>> consumer log for those patterns but couldn't find any in my logs.
>>>
>>> Can some one help me with this.
>>>
>>> Thanks
>>> Arjun Narasimha kota
>>>
>


Mime
View raw message