hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 牛兆捷 <nzjem...@gmail.com>
Subject Re: container memory resource request
Date Wed, 24 Apr 2013 00:14:45 GMT
Map task container:

2013-04-24 01:14:06,398 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000002 Container Transitioned from NEW to
ALLOCATED
2013-04-24 01:14:06,398 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hustnn
OPERATION=AM Allocated Container        TARGET=SchedulerApp
RESULT=SUCCESS  APPID=application_1366737158682_0002
 CONTAINERID=container_1366737158682_0002_01_000002
2013-04-24 01:14:06,398 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1366737158682_0002_01_000002 of capacity
<memory:2048, vCores:1> on host compute-0-0.local:44082, which currently
has 2 containers, <memory:4096, vCores:2> used and <memory:20480,
vCores:46> available
2013-04-24 01:14:06,400 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
assignedContainer application=application_1366737158682_0002
container=Container: [ContainerId: container_1366737158682_0002_01_000002,
NodeId: compute-0-0.local:44082, NodeHttpAddress: compute-0-0.local:8042,
Resource: <memory:2048, vCores:1>, Priority: 20, State: NEW, Token: null,
Status: container_id {, app_attempt_id {, application_id {, id: 2,
cluster_timestamp: 1366737158682, }, attemptId: 1, }, id: 2, }, state:
C_NEW, ] containerId=container_1366737158682_0002_01_000002 queue=default:
capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:2048,
vCores:1>usedCapacity=0.083333336, absoluteUsedCapacity=0.083333336,
numApps=1, numContainers=1 usedCapacity=0.083333336
absoluteUsedCapacity=0.083333336 used=<memory:2048, vCores:1>
cluster=<memory:24576, vCores:48>
2013-04-24 01:14:06,400 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting queues since queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:4096,
vCores:2>usedCapacity=0.16666667, absoluteUsedCapacity=0.16666667,
numApps=1, numContainers=2
2013-04-24 01:14:06,400 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.16666667
absoluteUsedCapacity=0.16666667 used=<memory:4096, vCores:2>
cluster=<memory:24576, vCores:48>
2013-04-24 01:14:07,015 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000002 Container Transitioned from
ALLOCATED to ACQUIRED
2013-04-24 01:14:07,405 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000002 Container Transitioned from ACQUIRED
to RUNNING
2013-04-24 01:14:13,920 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000002 Container Transitioned from RUNNING
to COMPLETED

reduce task container:
2013-04-24 01:14:14,923 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000003 Container Transitioned from NEW to
ALLOCATED
2013-04-24 01:14:14,923 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hustnn
OPERATION=AM Allocated Container        TARGET=SchedulerApp
RESULT=SUCCESS  APPID=application_1366737158682_0002
 CONTAINERID=container_1366737158682_0002_01_000003
2013-04-24 01:14:14,923 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1366737158682_0002_01_000003 of capacity
<memory:3072, vCores:1> on host compute-0-0.local:44082, which currently
has 2 containers, <memory:5120, vCores:2> used and <memory:19456,
vCores:46> available
2013-04-24 01:14:14,924 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
assignedContainer application=application_1366737158682_0002
container=Container: [ContainerId: container_1366737158682_0002_01_000003,
NodeId: compute-0-0.local:44082, NodeHttpAddress: compute-0-0.local:8042,
Resource: <memory:3072, vCores:1>, Priority: 10, State: NEW, Token: null,
Status: container_id {, app_attempt_id {, application_id {, id: 2,
cluster_timestamp: 1366737158682, }, attemptId: 1, }, id: 3, }, state:
C_NEW, ] containerId=container_1366737158682_0002_01_000003 queue=default:
capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:2048,
vCores:1>usedCapacity=0.083333336, absoluteUsedCapacity=0.083333336,
numApps=1, numContainers=1 usedCapacity=0.083333336
absoluteUsedCapacity=0.083333336 used=<memory:2048, vCores:1>
cluster=<memory:24576, vCores:48>
2013-04-24 01:14:14,924 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting queues since queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:5120,
vCores:2>usedCapacity=0.20833333, absoluteUsedCapacity=0.20833333,
numApps=1, numContainers=2
2013-04-24 01:14:14,924 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.20833333
absoluteUsedCapacity=0.20833333 used=<memory:5120, vCores:2>
cluster=<memory:24576, vCores:48>
2013-04-24 01:14:15,070 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000003 Container Transitioned from
ALLOCATED to ACQUIRED
2013-04-24 01:14:15,929 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000003 Container Transitioned from ACQUIRED
to RUNNING
2013-04-24 01:14:21,652 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000003 Container Transitioned from RUNNING
to COMPLETED

AM container:

2013-04-24 01:13:59,370 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000001 Container Transitioned from NEW to
ALLOCATED
2013-04-24 01:13:59,370 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hustnn
OPERATION=AM Allocated Container        TARGET=SchedulerApp
RESULT=SUCCESS  APPID=application_1366737158682_0002
 CONTAINERID=container_1366737158682_0002_01_000001
2013-04-24 01:13:59,370 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1366737158682_0002_01_000001 of capacity
<memory:2048, vCores:1> on host compute-0-0.local:44082, which currently
has 1 containers, <memory:2048, vCores:1> used and <memory:22528,
vCores:47> available
2013-04-24 01:13:59,374 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
assignedContainer application=application_1366737158682_0002
container=Container: [ContainerId: container_1366737158682_0002_01_000001,
NodeId: compute-0-0.local:44082, NodeHttpAddress: compute-0-0.local:8042,
Resource: <memory:2048, vCores:1>, Priority: 0, State: NEW, Token: null,
Status: container_id {, app_attempt_id {, application_id {, id: 2,
cluster_timestamp: 1366737158682, }, attemptId: 1, }, id: 1, }, state:
C_NEW, ] containerId=container_1366737158682_0002_01_000001 queue=default:
capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0,
vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1,
numContainers=0 usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0,
vCores:0> cluster=<memory:24576, vCores:48>
2013-04-24 01:13:59,374 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting queues since queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:2048,
vCores:1>usedCapacity=0.083333336, absoluteUsedCapacity=0.083333336,
numApps=1, numContainers=1
2013-04-24 01:13:59,374 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.083333336
absoluteUsedCapacity=0.083333336 used=<memory:2048, vCores:1>
cluster=<memory:24576, vCores:48>
2013-04-24 01:13:59,376 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000001 Container Transitioned from
ALLOCATED to ACQUIRED
2013-04-24 01:13:59,377 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Storing attempt: AppId: application_1366737158682_0002 AttemptId:
appattempt_1366737158682_0002_000001 MasterContainer: Container:
[ContainerId: container_1366737158682_0002_01_000001, NodeId:
compute-0-0.local:44082, NodeHttpAddress: compute-0-0.local:8042, Resource:
<memory:2048, vCores:1>, Priority: 0, State: NEW, Token: null, Status:
container_id {, app_attempt_id {, application_id {, id: 2,
cluster_timestamp: 1366737158682, }, attemptId: 1, }, id: 1, }, state:
C_NEW, ]
2013-04-24 01:13:59,379 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1366737158682_0002_000001 State change from SCHEDULED to
ALLOCATED_SAVING
2013-04-24 01:13:59,381 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
Storing info for attempt: appattempt_1366737158682_0002_000001
2013-04-24 01:13:59,383 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1366737158682_0002_000001 State change from ALLOCATED_SAVING to
ALLOCATED
2013-04-24 01:13:59,389 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
Launching masterappattempt_1366737158682_0002_000001
2013-04-24 01:13:59,414 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
Setting up container Container: [ContainerId:
container_1366737158682_0002_01_000001, NodeId: compute-0-0.local:44082,
NodeHttpAddress: compute-0-0.local:8042, Resource: <memory:2048, vCores:1>,
Priority: 0, State: NEW, Token: null, Status: container_id {,
app_attempt_id {, application_id {, id: 2, cluster_timestamp:
1366737158682, }, attemptId: 1, }, id: 1, }, state: C_NEW, ] for AM
appattempt_1366737158682_0002_000001
2013-04-24 01:13:59,414 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
Command to launch container container_1366737158682_0002_01_000001 :
$JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties
-Dyarn.app.mapreduce.container.log.dir=<LOG_DIR>
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
 -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2013-04-24 01:13:59,968 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done
launching container Container: [ContainerId:
container_1366737158682_0002_01_000001, NodeId: compute-0-0.local:44082,
NodeHttpAddress: compute-0-0.local:8042, Resource: <memory:2048, vCores:1>,
Priority: 0, State: NEW, Token: null, Status: container_id {,
app_attempt_id {, application_id {, id: 2, cluster_timestamp:
1366737158682, }, attemptId: 1, }, id: 1, }, state: C_NEW, ] for AM
appattempt_1366737158682_0002_000001
2013-04-24 01:13:59,968 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1366737158682_0002_000001 State change from ALLOCATED to LAUNCHED
2013-04-24 01:14:00,365 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1366737158682_0002_01_000001 Container Transitioned from ACQUIRED
to RUNNING


2013/4/24 Zhijie Shen <zshen@hortonworks.com>

> Would you please look into the resourcemanager log, and check how many
> containers are allocated and what the allocated memory is? You may want to
> search the log with "assignedContainer".
>
>
> On Tue, Apr 23, 2013 at 10:19 AM, 牛兆捷 <nzjemail@gmail.com> wrote:
>
> > I config them in mapred-site.xml like below, I set them less then 1000
> for
> > the normalization as you said:
> >
> > "
> > <property>
> >     <name>yarn.app.mapreduce.am.resource.mb</name>
> >     <value>900</value>
> >   </property>
> >   <property>
> >     <name>mapreduce.map.memory.mb</name>
> >     <value>900</value>
> >   </property>
> >   <property>
> >     <name>mapreduce.reduce.memory.mb</name>
> >     <value>900</value>
> >   </property>
> > "
> >
> > Then I run just one map, as you said there are 2 contained will be
> > launched, one for A/M master, the other for map task.
> > However, the 2 container cost 4G memory which I see from yarn UI
> interface.
> >
> >
> >
> >
> > 2013/4/24 Zhijie Shen <zshen@hortonworks.com>
> >
> > > Do you mean the memory assigned for the container of M/R's AM? Did you
> > set
> > > ContainerLaunchContext.setResource?
> > >
> > > AFAIK, by default, yarn.scheduler.minimum-allocation-mb = 1024 and
> > > yarn.app.mapreduce.am.resource.mb
> > > = 1536. So, M/R job will request 1536 for its AM, but Yarn's scheduler
> > will
> > > normalize the request to 2048, which is no less than 1536, and is
> > multiple
> > > times of the min allocation.
> > >
> > >
> > > On Tue, Apr 23, 2013 at 8:43 AM, 牛兆捷 <nzjemail@gmail.com> wrote:
> > >
> > > > I am using 2.0.3-alpha, I don't set the map memory capacity
> explicitly,
> > > > then "resourceCapacity.setMemory" should set the default memory
> request
> > > to
> > > > 1024mb,
> > > > However 2048 Memory is assigned to this container.
> > > >
> > > > Why it does like this?
> > > >
> > > > --
> > > > *Sincerely,*
> > > > *Zhaojie*
> > > > *
> > > > *
> > > >
> > >
> > >
> > >
> > > --
> > > Zhijie Shen
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> >
> >
> >
> > --
> > *Sincerely,*
> > *Zhaojie*
> > *
> > *
> >
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>



-- 
*Sincerely,*
*Zhaojie*
*
*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message