hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-1529) Add Localization overhead metrics to NM
Date Mon, 23 Dec 2013 22:45:51 GMT
Gera Shegalov created YARN-1529:

             Summary: Add Localization overhead metrics to NM
                 Key: YARN-1529
                 URL: https://issues.apache.org/jira/browse/YARN-1529
             Project: Hadoop YARN
          Issue Type: Bug
          Components: nodemanager
            Reporter: Gera Shegalov
            Assignee: Gera Shegalov

Users are often unaware of localization cost that their jobs incur. To measure effectiveness
of localization caches it is necessary to expose the overhead in the form of metrics.

We propose addition of the following metrics to NodeManagerMetrics.

When a container is about to launch, its set of LocalResources has to be fetched from a central
location, typically on HDFS, that results in a number of download requests for the files missing
in caches.

LocalizedFilesMissed: total files (requests) downloaded from DFS. Cache misses.

LocalizedFilesCached: total localization requests that were served from local caches. Cache

LocalizedBytesMissed: total bytes downloaded from DFS due to cache misses.

LocalizedBytesCached: total bytes satisfied from local caches.

Localized(Files|Bytes)CachedRatio: percentage of localized (files|bytes) that were served
out of cache: ratio = 100 * caches / (caches + misses)

LocalizationDownloadNanos: total elapsed time in nanoseconds for a container to go from ResourceRequestTransition
to LocalizedTransition

This message was sent by Atlassian JIRA

View raw message