hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jerson bejarano <jsbejara...@gmail.com>
Subject Incorporate job into subpools
Date Tue, 22 Jan 2019 19:21:25 GMT
Greetings,

I am writing in order to get advice on how to create a placement rule, that
incorporates jobs into subpools located at arbitrary levels of depth
automatically. The job allocation will still be performed based on either
the [username], [primaryGroup], or [secondaryGroup] tags.
However, I want to be able to allocate users in these pools usingarbitrary
depth levels of subpools, not just the first levels like root.[primaryGroup]
or root.[secondaryGroup].

For example, if I have two pools called 'root.analytics' and
'root.development'. And under these, a number of subpools for any number of
different user groups. Example:

{
           root {
                 analytics {
                            groupA,
                            groupB,
                            groupC
                                 },

                development {
                          groupD,
                          groupE


                                    }
               }
}

How can I create or configure a placement rule that will automaticaly allocate
users from groupA, groupB, and groupC, into their respective subpools under
root.analytics? Same for groupD and groupE under
root.development.

The available rules that perform allocation based on the [primaryGroup] and
[secondaryGroup] tags are relative to the 'root' pool. Which MEANS That If
I create the group pools at the first level of depth, then I Can perform
allocation automatically using the 'root.[primaryGroup]' rule.

However I don't see how to create a rule that will do what I want, which is
to place the users jobs automatically under 'root.analytics.[primaryGroup]'
or 'root.development.[primaryGroup]' instead.

In fact, if there is a way of doing it for arbitrary levels of depth, like
'root.level1.level2.level3.[primaryGroup]', then that would allow for a
much better classification and management of weights than having all
'groupX' pools at the first level under 'root'.

I am open to the idea of creating custom placement rules based on the Java
classes of the existing placement rules, however I am not sure that this is
in fact the only possible way to achieve this. Maybe the existing rules
support our use case in some way that I'm not aware of,
so that's why I'm asking for advice before making the decision.


Thank you for your consideration

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message