hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hung (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-6491) Move totalMB and totalVirtualCores computation from ClusterMetricsInfo to QueueMetrics
Date Wed, 19 Apr 2017 02:24:41 GMT
Jonathan Hung created YARN-6491:

             Summary: Move totalMB and totalVirtualCores computation from ClusterMetricsInfo
to QueueMetrics
                 Key: YARN-6491
                 URL: https://issues.apache.org/jira/browse/YARN-6491
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Jonathan Hung

Right now in ClusterMetricsInfo.java we do this:{noformat}    if (rs instanceof CapacityScheduler)
      this.totalMB = metrics.getTotalMB();
      this.totalVirtualCores = metrics.getTotalVirtualCores();
    } else {
      this.totalMB = availableMB + allocatedMB;
      this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores;

We'd like to have totalMB and totalVirtualCores as QueueMetrics fields. But since QueueMetrics
is scheduler agnostic we can't really just move this. It seems the way totalMB and totalVirtualCores
is computed across FS and CS should be standardized.

Right now CS does not include reservedMB in allocatedMB, while FS does (as far as I can tell).
At least in <= 2.7, when a container is reserved, queueUsage.getUsed is incremented, which
is the value used to determine if a queue can be assigned (AbstractCSQueue#canAssignToThisQueue).
So I think it makes sense to increment allocatedMB when a container is reserved, and not increment
it when a reserved container is allocated, to reflect the fact that if allocated + reserved
> queueLimit, allocation will fail, so the allocatedMB metric should also be > queueLimit.

Still not sure if the same is true >= 2.8. Would appreciate any input on this (or any of
the mentioned proposals).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message