kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun Kota <ar...@socialtwist.com>
Subject Re: High level consumer replaying the messages
Date Fri, 11 Apr 2014 00:52:14 GMT
I have 12 partitions on 3 different nodes.
On Apr 10, 2014 11:41 PM, "Maung Than" <maung_than@apple.com> wrote:

>
> >>> I have set up kafka 0.8 in 3 servers. I have pushed some data into
> these servers. The number of partitions i use is 12, with a replication
> factor of 2.
>
> Are you running multiple broker instances on a single server.
> Or your 12 partitions for multiple topics.
> I thought you should not have more partitions then the number of brokers
> in the cluster for a topic fro better load balancing and failover.
>
> Thanks,
> Maung
>
> On Apr 9, 2014, at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:
>
> > Hi ,
> >
> > I have set up kafka 0.8 in 3 servers. I have pushed some data into these
> servers. The number of partitions i use is 12, with a replication factor of
> 2.
> > We use high level consumer to consume messages. we set auto commit to
> false, and we do commits almost after each and every successful message
> consumption.
> > The problem i am facing now is, If i restart my consumers, then the
> messages are getting replayed.
> >
> > I checked the
> >
> > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
> --zkconnect zkhost:zkport --topic testtopic
> >
> > I see a high lag in some of the partitions and some partitions have none
> as the owner.
> > To verify consumer rebalance i used the following command
> >
> > bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group
> group1 --zookeeper.connect zkhost:zkport
> >
> > but i see errors
> >
> > [2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
> (kafka.tools.VerifyConsumerRebalance$)
> > [2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
> (kafka.tools.VerifyConsumerRebalance$)
> >
> >
> > I have seen this (
> http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieMj_eSy0YQUCoEMoxg@mail.gmail.com%3E)
> in the mail archives and i checked my consumer log for those patterns but
> couldn't find any in my logs.
> >
> > Can some one help me with this.
> >
> > Thanks
> > Arjun Narasimha kota
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message