kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: Monitoring offset lag
Date Thu, 07 Jul 2016 16:11:35 GMT
Hi Tom,
if you need a commercially proven lag monitoring solution (and all other
Kafka and ZK metrics) take a look at our SPM.
Hope you don't mind me plugging this one in :)

[image: Inline image 1]

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Wed, Jul 6, 2016 at 5:42 PM, Tom Dearman <tom.dearman@gmail.com> wrote:

> I recently had a problem on my production which I believe was a
> manifestation of the issue kafka-2978 (Topic partition is not sometimes
> consumed after rebalancing of consumer group), this is fixed in 0.9.0.1 and
> we will upgrade our client soon.  However, it made me realise that I didn’t
> have any monitoring set up on this.  The only thing I can find as a metric
> is the
> kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+),
> which, if I understand correctly, is the max lag of any partition that that
> particular consumer is consuming.
> 1. If I had been monitoring this, and if my consumer was suffering from
> the issue in kafka-2978, would I actually have been alerted, i.e. since the
> consumer would think it is consuming correctly would it not have updated
> the metric.
> 2. There is another way to see offset lag using the command
> /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server
> 10.10.1.61:9092 --describe —group consumer_group_name and parsing the
> response.  Is it safe or advisable to do this?  I like the fact that it
> tells me each partition lag, although it is also not available if no
> consumer from the group is currently consuming.
> 3. Is there a better way of doing this?

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message