hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com>
Subject Re: Host-specific ContainerRequest ignored
Date Fri, 16 May 2014 17:40:25 GMT
Hey Sandy,

I *believe* so, but I might have been on the FIFO scheduler when I did (or
using Capacity with a -1 node locality delay rather than 40).

In either case, I've backed off this pursuit. We really just want even
distribution of containers on NMs, which Fair Share is providing for us
out of the box, as you've suggested already. We were initially pursuing
baking this behavior into the AM, but backed off when we had difficulties
with the host assignment (as documented on YARN-2027). Eventually, we'll
probably adopt whatever anti-affinity based features YARN ends up
providing anyway--I saw Steve open up some JIRAs on this already.

Cheers,
Chris

On 5/15/14 12:21 AM, "Sandy Ryza" <sandy.ryza@cloudera.com> wrote:

>Hi Chris,
>
>Did you make sure to pass in the ContainerRequest rack param as null?
>
>-Sandy
>
>
>On Wed, May 7, 2014 at 12:44 PM, Chris Riccomini
><criccomini@linkedin.com>wrote:
>
>> Hey Guys,
>>
>> I am creating a container request:
>>
>>
>>   protected def requestContainers(memMb: Int, cpuCores: Int, containers:
>> Int) {
>>
>>     info("Requesting %d container(s) with %dmb of memory" format
>> (containers, memMb))
>>
>>     val capability = Records.newRecord(classOf[Resource])
>>
>>     val priority = Records.newRecord(classOf[Priority])
>>
>>     priority.setPriority(0)
>>
>>     capability.setMemory(memMb)
>>
>>     capability.setVirtualCores(cpuCores)
>>
>>     (0 until containers).foreach(idx => amClient.addContainerRequest(new
>> ContainerRequest(capability, null, null, priority)))
>>
>>   }
>>
>> This pretty closely mirrors the distributed shell example.
>>
>> If I put an array with a host string in the ContainerRequest, YARN seems
>> to completely ignore this request, and continues to put all containers
>>on
>> one or two nodes in the grid, which aren't the ones I requested, even
>> though the grid is completely empty, and there are 15 nodes available.
>>This
>> also holds true if I put "false" for relax locality. I'm running the
>> CapacityScheduler with a node-locality-delay set to 40. Previously, I
>>tried
>> the FifoScheduler, and it exhibited the same behavior.
>>
>> All NMs are just using the /default-rack for their rack. The strings
>>that
>> I'm putting in the hosts String[] parameter in ContainerRequest are hard
>> coded to exactly match the NodeIds being listed in the NMs.
>>
>> What am I doing wrong? I feel like I'm missing some configuration on the
>> capacity scheduler or NMs or something.
>>
>> Cheers,
>> Chris
>>


Mime
View raw message