hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: Node scheduling in 2.1.x
Date Fri, 06 Sep 2013 18:52:53 GMT
Have you taken a look at https://issues.apache.org/jira/browse/YARN-326 ? 

-- Hitesh 

On Sep 6, 2013, at 11:02 AM, hilfi alkaff wrote:

> Thanks for all the replies. I think I have found the relevant codes that I
> would like to modify. That said, a project that I'm doing now requires
> containers to have network bandwidth as one of its resources (In
> Resource.java: it currently only models memory).
> 
> Since I'm planning to implement it anyway, I hope to be able to help
> Hadoop's development. However, I could not find the relevant JIRA for this.
> If you know of an existing ticket that is relevant to the aforementioned
> issue, let me know. If there is none, should I make my changes first (as
> listed http://wiki.apache.org/hadoop/HowToContribute) and get back after
> I'm done with my code?
> 
> Thanks in advance.
> 
> 
> On Fri, Sep 6, 2013 at 6:37 AM, Steve Loughran <stevel@hortonworks.com>wrote:
> 
>> worth adding is that this can generate a bias towards affinitive
>> assignment of an apps containers; for the YARN-896 service we've put
>> anti-affinity as a subtask, along with having AM opt to get notifications
>> if assignments can't be met in a bounded period (or it could just examine
>> its queue of outstanding requests and reach the same conclusion based on
>> when the requests were submitted)
>> 
>> 
>> On 6 September 2013 07:43, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>> 
>>> That's right.  Nodes keep checking in and, when they do, the
>>> ResourceManager looks for outstanding requests.  This means that
>> assignment
>>> of containers to nodes depends on the order that they heartbeat in.  If
>>> container requests come in for specific nodes locality is achieved
>> through
>>> delay scheduling - the ResourceManager will wait for a configurable
>> number
>>> of heartbeats before assigning a container to a non-local node.  If
>> strict
>>> locality is turned on, the ResourceManager will wait indefinitely for a
>>> local node.
>>> 
>>> -Sandy
>>> 
>>> 
>>> On Fri, Sep 6, 2013 at 3:33 PM, hilfi alkaff <hilfialkaff@gmail.com>
>>> wrote:
>>> 
>>>> I see. What I'm wondering about is; when an application master tries to
>>>> request a container from resource manager, which part of the code in
>> the
>>>> resource manager actually decide which node to fetch this container
>> from.
>>>> Is this step being done asynchronously (ie: Nodes keep checking if
>> there
>>>> are requests from the ResourceManager during the node update event?)
>>>> 
>>>> 
>>>> On Fri, Sep 6, 2013 at 1:22 AM, Sandy Ryza <sandy.ryza@cloudera.com>
>>>> wrote:
>>>> 
>>>>> Hi Hilfi,
>>>>> 
>>>>> Nodes are constantly heartbeating to the ResourceManager.  A node
>>> update
>>>>> event is triggered each time this happens.
>>>>> 
>>>>> -Sandy
>>>>> 
>>>>> 
>>>>> On Fri, Sep 6, 2013 at 3:20 PM, hilfi alkaff <hilfialkaff@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I'm trying to trace the code flow on the scheduling done in YARN.
I
>>>> would
>>>>>> like to know where the code that does which node to schedule for
>> the
>>>>> jobs.
>>>>>> 
>>>>>> I found the handle() function in the resource manager's scheduler
>>> (eg:
>>>>>> CapacityScheduler.java) that handles node update event which then
>>>>> executes
>>>>>> the assignment of containers for that particular node, but I do not
>>>>>> understand how that node even get chosen.
>>>>>> 
>>>>>> If anybody could tell me about a file, function or module name that
>>>> does
>>>>>> this, that would be extremely helpful.
>>>>>> 
>>>>>> --
>>>>>> ~Hilfi Alkaff~
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> ~Hilfi Alkaff~
>>>> 
>>> 
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>> 
> 
> 
> 
> -- 
> ~Hilfi Alkaff~


Mime
View raw message