spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王晓雨 <wangxiao...@jd.com>
Subject Memory used in Spark-0.9.0-incubating
Date Thu, 25 Sep 2014 10:21:37 GMT
ENV:
Spark:0.9.0-incubating
Hadoop:2.3.0

I run spark task on Yarn. I see the log in Nodemanager:
2014-09-25 17:43:34,141 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

Memory usage of ProcessTree 549 for container-id 
container_1411635522254_0001_01_000005: *4.5 GB**of 5 GB* physical 
memory used; 5.0 GB of 10.5 GB virtual memory used
2014-09-25 17:43:37,171 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

Memory usage of ProcessTree 549 for container-id 
container_1411635522254_0001_01_000005: *4.5 GB**of 5 GB* physical 
memory used; 5.0 GB of 10.5 GB virtual memory used
2014-09-25 17:43:40,210 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

Memory usage of ProcessTree 549 for container-id 
container_1411635522254_0001_01_000005: *4.5 GB**of 5 GB* physical 
memory used; 5.0 GB of 10.5 GB virtual memory used
2014-09-25 17:43:43,239 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

Memory usage of ProcessTree 549 for container-id 
container_1411635522254_0001_01_000005: *4.5 GB of 5 GB* physical memory 
used; 5.0 GB of 10.5 GB virtual memory used
2014-09-25 17:43:46,269 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

Memory usage of ProcessTree 549 for container-id 
container_1411635522254_0001_01_000005: *4.5 GB of 5 GB* physical memory 
used; 5.0 GB of 10.5 GB virtual memory used

My task parameter is :
--num-workers 4 --master-memory 2g --worker-memory 4g --worker-cores 4
In myopinion "--worker-memory 4g" 4g is the maximum memory for container .
But why "4.5 GB of 5 GB physical memory used" in the log?
And where to config "5G" maxinum memory for container?

-- 


王晓雨
【云平台-开放云事业部】数据引擎组
-------------------------------------------------------------------------------------
手机:18600049984
电话:01062425760
邮箱:wangxiaoyu1@jd.com
邮编:100195
地址:北京市海淀区杏石口路65号益园文创基地C区11号楼3层
-------------------------------------------------------------------------------------


Mime
View raw message