hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13128) Manage Hadoop RPC resource usage via resource coupon
Date Mon, 03 Oct 2016 22:10:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543586#comment-15543586

Konstantin Shvachko commented on HADOOP-13128:

Will this also affect WebHDFS clients, or is it limited to RPCs only?
Http clients can be as "aggressive" as RPC ones based on my experience.

> Manage Hadoop RPC resource usage via resource coupon
> ----------------------------------------------------
>                 Key: HADOOP-13128
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13128
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Xiaoyu Yao
>            Assignee: Xiaoyu Yao
>         Attachments: HADOOP-13128-Proposal-20160511.pdf
> HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the
fairness usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently
manages the CPU and Memory resources for jobs/tasks but not the storage resources such as
HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may
send too many RPC requests to HDFS namenode and get demoted into low priority call queues
due to lack of reservation/coordination. 
> To better support multi-tenancy use cases like above, we propose to manage RPC server
resource usage via coupon mechanism integrated with YARN. The idea is to allow YARN request
HDFS storage resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode
on behalf of the job upon submission time.  Once granted, the tasks will include the coupon
identifier in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the
state of the coupon usage based on the scheduler policy (fairness or priority) to match the
RPC priority with the YARN scheduling priority.
> I will post a proposal with more detail shortly. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message