hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gaurav <gau...@datatorrent.com>
Subject Re: Allocating Containers on a particular Node in Yarn
Date Wed, 13 Nov 2013 23:44:36 GMT
Hi,

I had set the locality to false and request becomes as follows

/ {Priority: 0, Capability: <memory:8192, vCores:0>, # Containers: 1, 
Location: *, Relax Locality: false}, {Priority: 0, Capability: 
<memory:8192, vCores:0>, # Containers: 1, Location: /default-rack, Relax 
Locality: true}, {Priority: 0, Capability: <memory:8192, vCores:0>, # 
Containers: 1, Location: h1 Relax Locality: true}//
/
I checked AMRMClientImpl class and it is not respecting relaxlocality 
for node and racks (line 361 and368) resulting the above request.
Since one of ask has location set to h1, I expect container to be 
allocated on h1 but I get container on random hosts. I tweaked the code 
in AMRMClientImpl and made sure that relaxlocality is respected for node 
and rack, my new request becomes as following

/ {Priority: 0, Capability: <memory:8192, vCores:0>, # Containers: 1, 
Location: *, Relax Locality: false}, {Priority: 0, Capability: 
<memory:8192, vCores:0>, # Containers: 1, Location: /default-rack, Relax 
Locality: false}, {Priority: 0, Capability: <memory:8192, vCores:0>, # 
Containers: 1, Location: h1 Relax Locality: false}//
/
Now I get container on correct node h1. Not sure how will the fail over 
work in this case.

Thanks
-Gaurav

On 11/13/2013 3:23 PM, Alejandro Abdelnur wrote:
> Gaurav,
>
> Setting relaxLocality to FALSE should do it.
>
> thanks.
>
>
> On Wed, Nov 13, 2013 at 2:58 PM, gaurav <gaurav@datatorrent.com> wrote:
>
>>   Hi,
>>   I am trying to allocate containers on a particular node in Yarn but Yarn
>> is returning me containers on different node although the requested node
>> has resources available.
>>
>> I checked into the allocate(AllocateRequest request) function of
>> ApplicationMasterService and my request is as follows
>>
>> *request: ask { priority { priority: 1 } resource_name: "h2" capability {
>> memory: 1000 } num_containers: 2 } ask { priority { priority: 1 }
>> resource_name: "/default-rack" capability { memory: 1000 } num_containers:
>> 2 } ask { priority { priority: 1 } resource_name: "*" capability { memory:
>> 1000 } num_containers: 2 } response_id: 1 progress: 0.0*
>>
>> but the containers that I am getting back is as follows
>> [Container: [ContainerId: container_1384381084244_0001_01_000002, NodeId:
>> h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 }, ],
>> Container: [ContainerId: container_1384381084244_0001_01_000003, NodeId:
>> h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 }, ]]
>>
>> I am attaching the test case that I have written along with the mail. It
>> uses classes under org.apache.hadoop.yarn.server.resourcemanager package.
>>
>> Any pointers would be of great help
>>
>> Thanks
>> Gaurav
>>
>>
>>
>>
>>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message