kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Re: High level consumer replaying the messages
Date Thu, 10 Apr 2014 18:10:48 GMT

>>> I have set up kafka 0.8 in 3 servers. I have pushed some data into these servers.
The number of partitions i use is 12, with a replication factor of 2.

Are you running multiple broker instances on a single server. 
Or your 12 partitions for multiple topics. 
I thought you should not have more partitions then the number of brokers in the cluster for
a topic fro better load balancing and failover. 

Thanks,
Maung

On Apr 9, 2014, at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:

> Hi ,
> 
> I have set up kafka 0.8 in 3 servers. I have pushed some data into these servers. The
number of partitions i use is 12, with a replication factor of 2.
> We use high level consumer to consume messages. we set auto commit to false, and we do
commits almost after each and every successful message consumption.
> The problem i am facing now is, If i restart my consumers, then the messages are getting
replayed.
> 
> I checked the
> 
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect zkhost:zkport
--topic testtopic
> 
> I see a high lag in some of the partitions and some partitions have none as the owner.
> To verify consumer rebalance i used the following command
> 
> bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group group1 --zookeeper.connect
zkhost:zkport
> 
> but i see errors
> 
> [2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11] (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11] (kafka.tools.VerifyConsumerRebalance$)
> 
> 
> I have seen this ( http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieMj_eSy0YQUCoEMoxg@mail.gmail.com%3E)
in the mail archives and i checked my consumer log for those patterns but couldn't find any
in my logs.
> 
> Can some one help me with this.
> 
> Thanks
> Arjun Narasimha kota


Mime
View raw message