hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Node scheduling in 2.1.x
Date Fri, 06 Sep 2013 11:37:41 GMT
 worth adding is that this can generate a bias towards affinitive
assignment of an apps containers; for the YARN-896 service we've put
anti-affinity as a subtask, along with having AM opt to get notifications
if assignments can't be met in a bounded period (or it could just examine
its queue of outstanding requests and reach the same conclusion based on
when the requests were submitted)


On 6 September 2013 07:43, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

> That's right.  Nodes keep checking in and, when they do, the
> ResourceManager looks for outstanding requests.  This means that assignment
> of containers to nodes depends on the order that they heartbeat in.  If
> container requests come in for specific nodes locality is achieved through
> delay scheduling - the ResourceManager will wait for a configurable number
> of heartbeats before assigning a container to a non-local node.  If strict
> locality is turned on, the ResourceManager will wait indefinitely for a
> local node.
>
> -Sandy
>
>
> On Fri, Sep 6, 2013 at 3:33 PM, hilfi alkaff <hilfialkaff@gmail.com>
> wrote:
>
> > I see. What I'm wondering about is; when an application master tries to
> > request a container from resource manager, which part of the code in the
> > resource manager actually decide which node to fetch this container from.
> > Is this step being done asynchronously (ie: Nodes keep checking if there
> > are requests from the ResourceManager during the node update event?)
> >
> >
> > On Fri, Sep 6, 2013 at 1:22 AM, Sandy Ryza <sandy.ryza@cloudera.com>
> > wrote:
> >
> > > Hi Hilfi,
> > >
> > > Nodes are constantly heartbeating to the ResourceManager.  A node
> update
> > > event is triggered each time this happens.
> > >
> > > -Sandy
> > >
> > >
> > > On Fri, Sep 6, 2013 at 3:20 PM, hilfi alkaff <hilfialkaff@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm trying to trace the code flow on the scheduling done in YARN. I
> > would
> > > > like to know where the code that does which node to schedule for the
> > > jobs.
> > > >
> > > > I found the handle() function in the resource manager's scheduler
> (eg:
> > > > CapacityScheduler.java) that handles node update event which then
> > > executes
> > > > the assignment of containers for that particular node, but I do not
> > > > understand how that node even get chosen.
> > > >
> > > > If anybody could tell me about a file, function or module name that
> > does
> > > > this, that would be extremely helpful.
> > > >
> > > > --
> > > > ~Hilfi Alkaff~
> > > >
> > >
> >
> >
> >
> > --
> > ~Hilfi Alkaff~
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message