kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abu-Obeid, Osama" <Osama.Abu-Ob...@morganstanley.com>
Subject RE: consumer offset tool and JMX metrics do not match
Date Mon, 16 Nov 2015 16:27:10 GMT
I can observe the same thing:

- Lag values read through the Kafka consumer JMX is 0
- Lag values read through kafka-run-class.sh kafka.tools.ConsumerOffsetChecker is on average
200K-400K

When the Lag value in the Kafka consumer JMX is high (for example 5M), ConsumerOffsetChecker
shows a matching number.

I am running kafka_2.10-0.8.2.1

Osama

-----Original Message-----
From: allen chan [mailto:allen.michael.chan@gmail.com] 
Sent: Saturday, November 14, 2015 8:34 PM
To: users@kafka.apache.org
Subject: Re: consumer offset tool and JMX metrics do not match

I believe producers / brokers / and consumers has been restarted at different times.
What do you think the issue is?

On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbharaj@gmail.com>
wrote:

> Hi,
>
> Did any of your brokers/consumers undergo a restart between the time 
> after you had started the consumption and until you see this issue ?
>
> Thanks,
> Prabhjot
>
>
>
> On Sat, Nov 14, 2015 at 5:53 AM, allen chan 
> <allen.michael.chan@gmail.com>
> wrote:
>
> > I also looked at this metric in JMX and it is also 0
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
> ash*
> >
> > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> allen.michael.chan@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am comparing the output from kafka.tools.ConsumerOffsetChecker 
> > > vs JMX
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
> h,topic=logstash_fdm,partition=*)
> > > and they do not match.
> > >
> > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX 
> > > shows 0
> > for
> > > all partitions.
> > >
> > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> > >
> > > Has anyone seen this issue before? I am trying to use JMX to pull 
> > > stats into monitoring system
> > >
> > >
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand 
> binary, and those who don't"
>



--
Allen Michael Chan


--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies; do not disclose,
use or act upon the information; and notify the sender immediately. Mistransmission is not
intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the
extent permitted under applicable law, to monitor electronic communications. This message
is subject to terms available at the following link: http://www.morganstanley.com/disclaimers.
If you cannot access these links, please notify us by reply message and we will send the contents
to you. By messaging with Morgan Stanley you consent to the foregoing.
Mime
View raw message