hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured
Date Wed, 01 Nov 2017 22:41:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Erik Krogen updated HADOOP-15008:
    Attachment: HADOOP-15008.000.patch

Attaching v000 patch. The bug comes from the fact that the {{period}} being passed into the
{{MetricsSinkAdapter}} is in seconds, but the {{logicalTime}} passed in is milliseconds. The
sink adapter gets offered metrics on every period of the {{MetricsSystemImpl}} (where the
period is the GCD of all of the sinks' periods), and filters based on {{logicalTime % period
== 0}}. The mismatch in units would cause this filter to pass more often than it should.

> Metrics sinks may emit too frequently if multiple sink periods are configured
> -----------------------------------------------------------------------------
>                 Key: HADOOP-15008
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15008
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>            Priority: Minor
>         Attachments: HADOOP-15008.000.patch
> If there are multiple metrics sink periods configured, depending on what those periods
are, some sinks may emit too frequently. For example with the following:
> {code:title=hadoop-metrics2.properties}
> namenode.sink.file10.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file5.class=org.apache.hadoop.metrics2.sink.FileSink
> namenode.sink.file10.filename=namenode-metrics_per10.out
> namenode.sink.file5.filename=namenode-metrics_per5.out
> namenode.sink.file10.period=10
> namenode.sink.file5.period=5
> {code}
> I get the following:
> {code}
> ± for f in namenode-metrics_per*.out; do echo "$f" && grep "metricssystem.MetricsSystem"
$f | awk '{last=curr; curr=$1} END { print curr-last }'; done
> namenode-metrics_per10.out
> 5000
> namenode-metrics_per5.out
> 5000
> {code}
> i.e., for both metrics files, each record is 5000 ms apart, even though one of the sinks
has been configured to emit at 10s intervals

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message