hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Brennan (Jira)" <j...@apache.org>
Subject [jira] [Created] (YARN-10475) Scale RM-NM heartbeat interval based on node utilization
Date Tue, 27 Oct 2020 21:26:00 GMT
Jim Brennan created YARN-10475:
----------------------------------

             Summary: Scale RM-NM heartbeat interval based on node utilization
                 Key: YARN-10475
                 URL: https://issues.apache.org/jira/browse/YARN-10475
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: yarn
    Affects Versions: 2.10.1, 3.4.1
            Reporter: Jim Brennan
            Assignee: Jim Brennan


Add the ability to scale the RM-NM heartbeat interval based on node cpu utilization compared
to overall cluster cpu utilization.  If a node is over-utilized compared to the rest of the
cluster, it's heartbeat interval slows down.  If it is under-utilized compared to the rest
of the cluster, it's heartbeat interval speeds up.

This is a feature we have been running with internally in production for several years. 
It was developed by [~nroberts], based on the observation that larger faster nodes on our
cluster were under-utilized compared to smaller slower nodes. 

This feature is dependent on [YARN-10450], which added cluster-wide utilization metrics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message