kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject Re: High level consumer replaying the messages
Date Thu, 10 Apr 2014 12:12:05 GMT
A doubt regarding the same thing. If we are using the high level 
consumer and auto commit of offset is set to false, and we are not 
committing the offset for some reason how many messages can the consumer 
still read.

i mean if there are 100 messages in the kafka server, then I just 
started the consumer, how many messages can the consumer read, assuming 
after some time as the offset is not commited, consumer will not be able 
to consume any messages.

Thanks
Arjun NArasimha Kota

On Thursday 10 April 2014 09:13 AM, Jun Rao wrote:
> Do you see many rebalances in the consumer log? If so, see
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> ?
>
> Thanks,
>
> Jun
>
>
> On Wed, Apr 9, 2014 at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:
>
>> Hi ,
>>
>> I have set up kafka 0.8 in 3 servers. I have pushed some data into these
>> servers. The number of partitions i use is 12, with a replication factor of
>> 2.
>> We use high level consumer to consume messages. we set auto commit to
>> false, and we do commits almost after each and every successful message
>> consumption.
>> The problem i am facing now is, If i restart my consumers, then the
>> messages are getting replayed.
>>
>> I checked the
>>
>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
>> --zkconnect zkhost:zkport --topic testtopic
>>
>> I see a high lag in some of the partitions and some partitions have none
>> as the owner.
>> To verify consumer rebalance i used the following command
>>
>> bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group group1
>> --zookeeper.connect zkhost:zkport
>>
>> but i see errors
>>
>> [2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
>> (kafka.tools.VerifyConsumerRebalance$)
>> [2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
>> (kafka.tools.VerifyConsumerRebalance$)
>>
>>
>> I have seen this ( http://mail-archives.apache.org/mod_mbox/kafka-users/
>> 201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM
>> j_eSy0YQUCoEMoxg@mail.gmail.com%3E) in the mail archives and i checked my
>> consumer log for those patterns but couldn't find any in my logs.
>>
>> Can some one help me with this.
>>
>> Thanks
>> Arjun Narasimha kota
>>


Mime
View raw message