hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Almeida <almeida...@gmail.com>
Subject What's the best way to access the app configuration?
Date Mon, 19 Oct 2020 11:06:14 GMT
Hello,

I've been running a yarn cluster for the past couple years. Over time, my
users and I have had some ideas that could improve the scheduling behaviour
for us. Therefore, I decided to try them out by modifying the code. I've
made a few promising results with some small changes I made. Nothing really
good enough for sending upstream yet. One of the problems with my changes
being that I still haven't figured out how to access a configuration object
from the class where I need it.

For example, if I want to make changes to
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.ComputeFairShares
which are controllable via configuration, the only way I can think of would
be adding extra parameters to computeShares, which, in turn would require
instantiating FairSharePolicy and DominantResourceFairnessPolicy with extra
parameters and keep going, until I finally reach
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler,
which does finally parse the configuration file and would know of new
attributes I add. I was hoping I could have the configuration injected into
any class or at least a FairSchedulerConfiguration.getInstance() static
method. However, I don't think either exists.

Do you have any tips on this for me?

Cheers,
Rafael

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message