hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szilard Nemeth (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-9457) Integrate custom resource metrics better for FairScheduler
Date Sun, 07 Apr 2019 08:20:00 GMT
Szilard Nemeth created YARN-9457:

             Summary: Integrate custom resource metrics better for FairScheduler
                 Key: YARN-9457
                 URL: https://issues.apache.org/jira/browse/YARN-9457
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Szilard Nemeth

YARN-8842 added org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsForCustomResources.
This class stores all metrics data for custom resource types.
A field is there in QueueMetrics to hold an object of this class.

Similarly, YARN-9322 added FSQueueMetricsForCustomResources and added an object of this class
to FSQueueMetrics.

This jira is about to investigate how it is possible to integrate QueueMetricsForCustomResources
into QueueMetrics and FSQueueMetricsForCustomResources into FSQueueMetrics. 
The trick is that the Metrics annotation (org.apache.hadoop.metrics2.annotation.Metric) is
used to expose values on JMX.

We need to implement a mechanism where QueueMetrics / FSQueueMetrics classes do contain a
field of the custom resource values which is a map of resource names as keys, and longs as
This way, we don't need the new classes (QueueMetricsForCustomResources and FSQueueMetricsForCustomResources),
the code could be much cleaner and consistent.

The hardest part possibly is to find a way to expose metrics values from a map. We obviously
can't use the Metrics annotation so a mechanism is required to expose the values on JMX.
For a quick search, I haven't found any way like this in the code
[~wilfreds]: Are you aware of any way to expose values like this?
Most probably, we need to check how the Metrics annotation is processed, understand the whole
flow and check what is the underlying mechanism of the metrics propagation to the JMX interface.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message