kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: High level consumer replaying the messages
Date Fri, 11 Apr 2014 00:49:26 GMT
You don't need to match the number of partitions. Your nine consumers
should distribute those 12 partitions amongst themselves. Are you by
any chance consuming across a high latency link? Can you see if there
are rebalance failures in your consumer logs?

On Thu, Apr 10, 2014 at 12:27:39PM +0530, Arjun wrote:
> My consumer configuration has auto.offset.reset=smallest will this
> be causing the problem for replaying the messages?
> I got to know why some partitions have no owners. It was my mistake
> i have 12 partitions but i started only 9 consumers. So when i check
> the Consumer Offset checker tool 3 partitions were showing "none" as
> the owner.
> 
> If some one can please help why we get this
> "
> [kafka.utils.ZkUtils$] conflict in
> /consumers/group1/owners/testtopic/2 data:
> group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data:
> group1_ip-10-168-47-114-1397034341089-3861c6cc-1
> "
> and the replay of messages i would be grateful
> 
> Thanks
> Arjun Narasimha Kota
> 
> 
> On Thursday 10 April 2014 10:17 AM, Arjun wrote:
> >I see this a lot in the consumer logs
> > [kafka.utils.ZkUtils$] conflict in
> >/consumers/group1/owners/testtopic/2 data:
> >group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data:
> >group1_ip-10-168-47-114-1397034341089-3861c6cc-1
> >
> >I am not seeing any "expired" in my consumer logs
> >
> >Thanks
> >Arjun Narasimha Kota
> >
> >On Thursday 10 April 2014 09:13 AM, Jun Rao wrote:
> >>Do you see many rebalances in the consumer log? If so, see
> >>https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> >>
> >>?
> >>
> >>Thanks,
> >>
> >>Jun
> >>
> >>
> >>On Wed, Apr 9, 2014 at 8:38 PM, Arjun <arjun@socialtwist.com> wrote:
> >>
> >>>Hi ,
> >>>
> >>>I have set up kafka 0.8 in 3 servers. I have pushed some data
> >>>into these
> >>>servers. The number of partitions i use is 12, with a
> >>>replication factor of
> >>>2.
> >>>We use high level consumer to consume messages. we set auto commit to
> >>>false, and we do commits almost after each and every successful message
> >>>consumption.
> >>>The problem i am facing now is, If i restart my consumers, then the
> >>>messages are getting replayed.
> >>>
> >>>I checked the
> >>>
> >>>bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
> >>>--zkconnect zkhost:zkport --topic testtopic
> >>>
> >>>I see a high lag in some of the partitions and some partitions
> >>>have none
> >>>as the owner.
> >>>To verify consumer rebalance i used the following command
> >>>
> >>>bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance
> >>>--group group1
> >>>--zookeeper.connect zkhost:zkport
> >>>
> >>>but i see errors
> >>>
> >>>[2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>[2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
> >>>(kafka.tools.VerifyConsumerRebalance$)
> >>>
> >>>
> >>>I have seen this (
> >>>http://mail-archives.apache.org/mod_mbox/kafka-users/
> >>>201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM
> >>>j_eSy0YQUCoEMoxg@mail.gmail.com%3E) in the mail archives and i
> >>>checked my
> >>>consumer log for those patterns but couldn't find any in my logs.
> >>>
> >>>Can some one help me with this.
> >>>
> >>>Thanks
> >>>Arjun Narasimha kota
> >>>
> >
> 


Mime
View raw message