hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Suh (Jira)" <j...@apache.org>
Subject [jira] [Created] (YARN-10857) YarnClient Caching Addresses
Date Fri, 16 Jul 2021 20:14:00 GMT
Steve Suh created YARN-10857:

             Summary: YarnClient Caching Addresses
                 Key: YARN-10857
                 URL: https://issues.apache.org/jira/browse/YARN-10857
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: client, yarn
            Reporter: Steve Suh

We have noticed that when the YarnClient is initialized and used, it is not very resilient
when dns or /etc/hosts is modified in the following scenario:

Take for instance the following (and reproducable) sequence of events that can occur on a
service that instantiates and uses YarnClient. 
  - Yarn has rm HA enabled (`yarn.resourcemanager.ha.enabled` is `true`) and there are two
rms (rm1 and rm2).
  - `yarn.client.failover-proxy-provider` is set to `org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider`

1)	rm2 is currently the active rm
2)	/etc/hosts (or dns) is missing host information for rm2
3)	A service is started and it initializes the YarnClient at startup.
4)	At some point in time after YarnClient is done initializing, /etc/hosts is updated and
contains host information for rm2
5)	Yarn is queried using YarnClient, for instance calling `.getApplications()`
6)	All YarnClient attempts to communicate with rm2 fail with UnknownHostExceptions, even though
/etc/hosts now contains host information for it.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message