kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0
Date Tue, 27 Jan 2015 08:52:34 GMT
Is it actually getting double-counted? I tried reproducing this
locally but the BrokerTopicMetrics.Count lines up with the sum of the
PerTopic.Counts for various metrics.

On Tue, Jan 27, 2015 at 03:29:37AM -0500, Jason Rosenberg wrote:
> Ok,
> 
> It looks like the yammer MetricName is not being created correctly for the
> sub metrics that include a topic. E.g. a metric with an mbeanName like:
> 
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
> 
> appears to be malformed. A yammer MetricName has 4 fields that are used in
> creating a graphite metric, that are included in the constructor:
> group, type, name, scope.
> 
> In this case, the metric with the above mbeanName has these fields set in
> the MetricName:
> 
> group: "kafka.server"
> type: "BrokerTopicMetrics"
> name: "BytesInPerSec"
> scope: null
> 
> Thus, the topic metrics all look the same, and get lumped into the
> top-level BrokerTopicMetrics (and thus that will now be double counted). It
> looks like the fix for kafka-1481 was where things got broken. It seems to
> have introduced ‘tags’ in the building of metric names, and then those tags
> only get applied to the mbeanName, but get excluded from the metric name:
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
> 
> This is a pretty severe issue, since the yammer metrics for these stats
> will be double counted in aggregate, and the per-topic stats will be
> removed.
> 
> I should note too, in my previous email, I thought that only the per-topic
> BrokerTopicMetrics were missing, but also several other per-topic metrics
> are missing too, e.g. under kafka.log, etc.
> 
> Jason
> ​
> 
> On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <jbr@squareup.com> wrote:
> 
> > I can confirm that the per topic metrics are not coming through to the
> > yammer metrics registry.  I do see them in jmx (via jconsole), but the
> > MetricsRegistry does not have them.
> > All the other metrics are coming through that appear in jmx.
> >
> > This is with single node instance running locally.
> >
> > Jason
> >
> >
> >
> > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <kumar@nmsworks.co.in>
> > wrote:
> >
> >> If you are using multi-node cluster, then metrics may be reported from
> >> other servers.
> >> pl check all the servers in the cluster.
> >>
> >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <kylebanker@gmail.com>
> >> wrote:
> >>
> >> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> >> messages in
> >> > and out for all topics together and for each individual topic.
> >> >
> >> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> >> >
> >> > I'm only seeing the following:
> >> > BrokerTopicMetrics
> >> > - BytesInPerSec
> >> > - BytesOutPerSec
> >> > - BytesRejectedPerSec
> >> > - MessagesInPerSec
> >> >
> >> > What's more, despite lots of successful writes to the cluster, the
> >> values
> >> > for these remaining metrics are all zero.
> >> >
> >> > I saw that there was some refactoring of metric naming code. Was the
> >> > behavior supposed to have changed?
> >> >
> >> > Many thanks in advance.
> >> >
> >>
> >
> >


Mime
View raw message