kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benson Chen <Benson.C...@sas.com>
Subject RE: MBean for records-lag-max metric always 1.0 or -Infinity
Date Tue, 23 May 2017 20:34:46 GMT
We just tried upgrading the jar to 10.2 version and it works!  Also, I was misleading with
the prior version.  We were running 10.0.1 KafkaServer but using the 10.1 client jar.  With
the 10.2 client jar working against the 10.0.1 KafkaServer, the records-lag-max metrics is
now reporting correctly!
Anyway, it may be useful to document this for other users the compatibility of this very useful
metric.  Thanks.
From: Benson Chen
Sent: Tuesday, May 23, 2017 3:11 PM
To: users@kafka.apache.org
Cc: Hongjie Xin <Hongjie.Xin@sas.com>
Subject: MBean for records-lag-max metric always 1.0 or -Infinity

Hello,

We're researching autoscaling our microservices consuming from a Kafka Topic based on the
records-lag-max metric.  For the following MBean object:

 MBeans TAB: kafka.consumer-> consumer-fetcher-manager-metric -> consumer-1 -> Attributes
-> records-lag-max

In JConsole, we noticed that we consistently get either 1.0 or -Infinity.  We also implemented
in our Java code to invoke the metrics() method of KafkaConsumer.java to fetch the metrics
after a poll of records from the Topic.
Map<MetricName,? extends Metric> metrics()    //Get the metrics kept by the consumer
Printing out this record also shows 1.0 or -Infinity.
In our test we have a Kafka Producer sending in about 50 messages/s while a Kafka Consumer
is consuming about 20 messages/s.  We know that the Kafka Consumer is definitely lagging behind
because it continues to consume messages well after shutting down the Kafka Producer.
Is this a known bug to get the correct value for records-lag-max or do we need to tune the
configuration settings somehow?
We're running 0.10.1.1.
Thanks for your help.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message