kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Banker <kyleban...@gmail.com>
Subject Re: Consumer Group Lag Reporting
Date Tue, 07 Apr 2015 15:55:36 GMT
Thanks, Otis. I actually already have a reporting and alerting
infrastructure.

I mainly wanted to confirm that parsing the output of the offset checker is
the recommended practice for reporting consumer group offsets. Is this the
case? If so, I wanted to find out if any work is under way to make this
more streamlined.

On Mon, Apr 6, 2015 at 10:05 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> We track our Consumer Lag and all other Kafka metrics with SPM
> <http://sematext.com/spm> whose agent collects this info from Kafka
> Consumers.  Here's a chart of that:
> https://apps.sematext.com/spm-reports/s/mcBHvsXwgj -- this public chart
> doesn't show all filtering option, but you can filter lag by host, client
> id, topic, and partition.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Apr 6, 2015 at 6:14 PM, Kyle Banker <kylebanker@gmail.com> wrote:
>
> > What is the best practice for reporting the lag on individual consumer
> > groups (e.g., to Graphite)?
> >
> > A recent form post (http://search-hadoop.com/m/4TaT4x9qWm1) seems to
> > indicate that parsing the output of the consumer offset checker tool and
> > reporting that independently is what folks do. Is there a better way?
> >
> > Are there any plans for the Kafka brokers themselves to do this reporting
> > (e.g., the master for each partition reports lag for all of its consumer
> > groups)?
> >
> > Many thanks in advance.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message