hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Multiple resource requests for a given node (or all nodes)?
Date Tue, 13 Dec 2011 06:42:07 GMT
I'd argue that Robert is complaining that the interface *is not* MR-centric enough.

IAC, priorities is fairly generic. MR AM uses it to get constraints to stick.


On Dec 12, 2011, at 7:50 PM, Patrick Wendell wrote:

> Todd - that's a good question and I haven't looked closely into
> whether simply adding a multimap is enough or if there are more deeply
> seeded issues (at least to address this specific case). If it's the
> former I'll probably just submit a patch.
> Arun - that seems like a hack but I guess it is a sufficient
> workaround for current applications.
> I'm finishing up a bare-bones version of the Fair Scheduler right now
> (going to throw something up for review soon) but I haven't yet added
> preemption.  How this is going to work well with various types of
> applications is unclear. In the MR case we can probably just preempt
> based on priorities, since they are essentially just ordering
> constraints right now. As Robert points out, this interface is very
> MR-Centric right now - i'm not sure this generalizes well to other
> applications depending on how they use priorities.
> - Patrick
> On Mon, Dec 12, 2011 at 1:27 PM, Arun C Murthy <acm@hortonworks.com> wrote:
>> Use priorities to ask for different resource types.
>> Arun
>> On Dec 10, 2011, at 12:23 PM, Patrick Wendell wrote:
>>> If you look at how resource requests are stored now, they use a map
>>> keyed on the node hostname.
>>> == AppSchedulingInfo.java ==
>>>  final Map<Priority, Map<String, ResourceRequest>> requests =
>>>    new HashMap<Priority, Map<String, ResourceRequest>>();
>>> ========
>>> What happens if an application wants to request multiple container
>>> types on a given node. E.g. say I need 10 2GB containers and 10 1GB
>>> containers, and I don't care which node they are on (i.e. RMNode.ANY).
>>> I really want to store 2 resource requests under RMNode.ANY in this
>>> case... don't I?
>>> Is the model just that an AM would ask for these in series?
>>> - Patrick

View raw message