hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Multiple resource requests for a given node (or all nodes)?
Date Tue, 13 Dec 2011 03:50:45 GMT
Todd - that's a good question and I haven't looked closely into
whether simply adding a multimap is enough or if there are more deeply
seeded issues (at least to address this specific case). If it's the
former I'll probably just submit a patch.

Arun - that seems like a hack but I guess it is a sufficient
workaround for current applications.

I'm finishing up a bare-bones version of the Fair Scheduler right now
(going to throw something up for review soon) but I haven't yet added
preemption.  How this is going to work well with various types of
applications is unclear. In the MR case we can probably just preempt
based on priorities, since they are essentially just ordering
constraints right now. As Robert points out, this interface is very
MR-Centric right now - i'm not sure this generalizes well to other
applications depending on how they use priorities.

- Patrick

On Mon, Dec 12, 2011 at 1:27 PM, Arun C Murthy <acm@hortonworks.com> wrote:
> Use priorities to ask for different resource types.
> Arun
> On Dec 10, 2011, at 12:23 PM, Patrick Wendell wrote:
>> If you look at how resource requests are stored now, they use a map
>> keyed on the node hostname.
>> == AppSchedulingInfo.java ==
>>  final Map<Priority, Map<String, ResourceRequest>> requests =
>>    new HashMap<Priority, Map<String, ResourceRequest>>();
>> ========
>> What happens if an application wants to request multiple container
>> types on a given node. E.g. say I need 10 2GB containers and 10 1GB
>> containers, and I don't care which node they are on (i.e. RMNode.ANY).
>> I really want to store 2 resource requests under RMNode.ANY in this
>> case... don't I?
>> Is the model just that an AM would ask for these in series?
>> - Patrick

View raw message