hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13128) Manage Hadoop RPC resource usage via resource coupon
Date Wed, 11 May 2016 00:16:12 GMT
Xiaoyu Yao created HADOOP-13128:

             Summary: Manage Hadoop RPC resource usage via resource coupon
                 Key: HADOOP-13128
                 URL: https://issues.apache.org/jira/browse/HADOOP-13128
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Xiaoyu Yao
            Assignee: Xiaoyu Yao

HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the fairness
usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently
manages the CPU and Memory resources for jobs/tasks but not the storage resources such as
HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may
send too many RPC requests to HDFS namenode call queue and get demoted into low priority namenode
call queue due to lack of coordination. 

To better support multi-tenancy use cases like above, we propose to manage RPC server resource
usage via coupon mechanism integrated with YARN. The idea is to allow YARN request HDFS storage
resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode on behalf
of the job upon submission time.  Once granted, the tasks will include the coupon identifier
in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the state of
the coupon usage based on the scheduler policy (fairness or priority) to match the RPC priority
with the YARN scheduling priority. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message