kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sybrandy, Casey" <Casey.Sybra...@Six3Systems.com>
Subject RE: Consumers re-reading data issue
Date Thu, 13 Dec 2012 16:14:35 GMT
The server is version 3.3.4.  As for the client, I'm just using whatever Kakfa 0.7.0 came with.
 Not sure if that makes a different.  If it does, I guess upgrading is the way to go.

As for the GC part, I'll have to look into that some more.

Thanks.

-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Thursday, December 13, 2012 10:49 AM
To: users@kafka.apache.org
Subject: Re: Consumers re-reading data issue

What's the ZK version? You should use 3.3.4 or above, which fixed some bugs that could cause
rebalance to fail.

Also, a typical reason for many rebalancing is the consumer side GC. If so, you will see logs
saying sth like "expired session" for ZK.

Thanks,

Jun

On Thu, Dec 13, 2012 at 5:54 AM, Sybrandy, Casey < Casey.Sybrandy@six3systems.com> wrote:

> Jun,
>
> Funny you should ask...more data was loaded last night and no data was 
> sent to the brokers since Dec. 11.  The broker that I thought was the 
> issue has not appeared since I removed the topic from it, so I'm 
> guessing that's not the problem.
>
> Some other info that may be useful:
>
> We have over 1300 messages about rebalancing in our logs starting at 
> about
> 9:00 yesterday morning, so almost 24 hours of log data.  Granted, all 
> I did was a "grep rebalance <logfile> | wc -l", but that still seems 
> high.  Of these, there are 61 messages stating that it can't rebalance.
>
> Also, we've seen the following error message 244 times:
> org.I0Itec.zkclient.exception.ZkNoNodeException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode 
> = NoNode for
>
> Would upgrading to 0.7.1/0.7.2 resolve some/all of these?  Or at least 
> help?  I ask because I had to compile notes for a potential upgrade 
> and I noticed that there were several changes regarding rebalancing 
> and duplicate messages.
>
> Thanks.
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, December 13, 2012 1:03 AM
> To: users@kafka.apache.org
> Subject: Re: Consumers re-reading data issue
>
> Casey,
>
> Not sure what's happening there. Is this reproducible?
>
> Thanks,
>
> Jun
>
> On Wed, Dec 12, 2012 at 9:52 AM, Sybrandy, Casey < 
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Hello,
> >
> > We have a strange issue on one of our systems that we just can't 
> > seem to figure out.  We loaded in 5000 records yesterday and have 
> > not loaded any since then into our brokers.  Examining the logs for 
> > the brokers, we confirmed that no data has been sent to those topics 
> > since we did the load yesterday.  However, since the data load 
> > yesterday, we've had the same record show up in our consumer logs 5 times.
> >
> > Offsets have been checked (We log everything INFO and above by
> > default) and it doesn't look like that the consumers are reading 
> > from an older offset.  The only thing that I see that may cause this 
> > issue is a broker that sometimes is accessed by the consumers, but 
> > not all of the time.  It's an older broker that we don't use anymore 
> > for our project, but is still there for another project.  Until this 
> > morning, it did have the topic we were reading from on the broker.  
> > I did remove the topic by deleting the directories and restarting 
> > the broker, so hopefully the consumers don't try to use it again.
> >
> > Could that be why we're seeing the duplicate records?  Or is there 
> > something else we need to check for?
> >
> > Casey
> >
> >
>

Mime
View raw message