kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun <ar...@socialtwist.com>
Subject High level consumer replaying the messages
Date Thu, 10 Apr 2014 03:38:36 GMT
Hi ,

I have set up kafka 0.8 in 3 servers. I have pushed some data into these 
servers. The number of partitions i use is 12, with a replication factor 
of 2.
We use high level consumer to consume messages. we set auto commit to 
false, and we do commits almost after each and every successful message 
consumption.
The problem i am facing now is, If i restart my consumers, then the 
messages are getting replayed.

I checked the

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1 
--zkconnect zkhost:zkport --topic testtopic

I see a high lag in some of the partitions and some partitions have none 
as the owner.
To verify consumer rebalance i used the following command

bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group 
group1 --zookeeper.connect zkhost:zkport

but i see errors

[2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11] 
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11] 
(kafka.tools.VerifyConsumerRebalance$)


I have seen this ( 
http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieMj_eSy0YQUCoEMoxg@mail.gmail.com%3E)

in the mail archives and i checked my consumer log for those patterns 
but couldn't find any in my logs.

Can some one help me with this.

Thanks
Arjun Narasimha kota

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message