hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jagadish (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-8034) Clarification on preferredHost request with relaxedLocality
Date Fri, 16 Mar 2018 03:01:00 GMT
Jagadish created YARN-8034:

             Summary: Clarification on preferredHost request with relaxedLocality
                 Key: YARN-8034
                 URL: https://issues.apache.org/jira/browse/YARN-8034
             Project: Hadoop YARN
          Issue Type: Bug
            Reporter: Jagadish

I work on Apache Samza, a stateful stream-processing framework that leverages Yarn for resource
management. The Samza AM requests resources on specific hosts to schedule stateful jobs. We
set relaxLocality = true in these requests we make to Yarn. Often we have observed that we
don't get containers on the hosts that we requested them on and the Yarn RM returns containers
on arbitrary hosts. 

Do you know what the behavior of the FairScheduler/CapacityScheduler is when setting "relaxLocality
= true".I did play around by setting a high value for yarn.scheduler.capacity.node-locality-delay
but it did not seem to matter. However, when setting relaxLocality = false, we get resources
on the exact hosts we requested on.

The behavior I want from Yarn is "Honor locality to the best possible extent and only return
a container on an arbitrary host if the requested host is down". Is there a way to accomplish
If you can point me to the Scheduler code, I'm happy to look at it as well. For context, we
have continuous scheduling enabled in our clusters.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message