hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-7399) Yarn services metadata storage improvement
Date Thu, 26 Oct 2017 20:12:01 GMT
Eric Yang created YARN-7399:

             Summary: Yarn services metadata storage improvement
                 Key: YARN-7399
                 URL: https://issues.apache.org/jira/browse/YARN-7399
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: yarn-native-services
            Reporter: Eric Yang

In Slider, metadata is stored in user's home directory. Slider command line interface interacts
with HDFS directly to list deployed applications and invoke YARN API or HDFS API to provide
information to user. This design works for a single user manage his/her own applications.
When this design has been ported to Yarn services, it becomes apparent that this design is
difficult to list all deployed applications on Hadoop cluster for administrator to manage
applications. Resource Manager needs to crawl through every user's home directory to compile
metadata about deployed applications. This can trigger high load on namenode to list hundreds
or thousands of list directory calls owned by different users. Hence, it might be best to
centralize the metadata storage to Solr or HBase to reduce number of IO calls to namenode
for manage applications.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message