kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: High level consumer replaying the messages
Date Thu, 10 Apr 2014 03:43:12 GMT
Do you see many rebalances in the consumer log? If so, see
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
?

Thanks,

Jun


On Wed, Apr 9, 2014 at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:

> Hi ,
>
> I have set up kafka 0.8 in 3 servers. I have pushed some data into these
> servers. The number of partitions i use is 12, with a replication factor of
> 2.
> We use high level consumer to consume messages. we set auto commit to
> false, and we do commits almost after each and every successful message
> consumption.
> The problem i am facing now is, If i restart my consumers, then the
> messages are getting replayed.
>
> I checked the
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
> --zkconnect zkhost:zkport --topic testtopic
>
> I see a high lag in some of the partitions and some partitions have none
> as the owner.
> To verify consumer rebalance i used the following command
>
> bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group group1
> --zookeeper.connect zkhost:zkport
>
> but i see errors
>
> [2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
> (kafka.tools.VerifyConsumerRebalance$)
> [2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
> (kafka.tools.VerifyConsumerRebalance$)
>
>
> I have seen this ( http://mail-archives.apache.org/mod_mbox/kafka-users/
> 201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM
> j_eSy0YQUCoEMoxg@mail.gmail.com%3E) in the mail archives and i checked my
> consumer log for those patterns but couldn't find any in my logs.
>
> Can some one help me with this.
>
> Thanks
> Arjun Narasimha kota
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message