Jonathan Hung created YARN-6491:
-----------------------------------
Summary: Move totalMB and totalVirtualCores computation from ClusterMetricsInfo
to QueueMetrics
Key: YARN-6491
URL: https://issues.apache.org/jira/browse/YARN-6491
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jonathan Hung
Right now in ClusterMetricsInfo.java we do this:{noformat} if (rs instanceof CapacityScheduler)
{
this.totalMB = metrics.getTotalMB();
this.totalVirtualCores = metrics.getTotalVirtualCores();
} else {
this.totalMB = availableMB + allocatedMB;
this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores;
}{noformat}
We'd like to have totalMB and totalVirtualCores as QueueMetrics fields. But since QueueMetrics
is scheduler agnostic we can't really just move this. It seems the way totalMB and totalVirtualCores
is computed across FS and CS should be standardized.
Right now CS does not include reservedMB in allocatedMB, while FS does (as far as I can tell).
At least in <= 2.7, when a container is reserved, queueUsage.getUsed is incremented, which
is the value used to determine if a queue can be assigned (AbstractCSQueue#canAssignToThisQueue).
So I think it makes sense to increment allocatedMB when a container is reserved, and not increment
it when a reserved container is allocated, to reflect the fact that if allocated + reserved
> queueLimit, allocation will fail, so the allocatedMB metric should also be > queueLimit.
Still not sure if the same is true >= 2.8. Would appreciate any input on this (or any of
the mentioned proposals).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
|