spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王晓雨 <wangxiao...@jd.com>
Subject Re: Memory used in Spark-0.9.0-incubating
Date Thu, 25 Sep 2014 10:41:36 GMT
My yarn-site.xml config:
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>16384</value>
</property>

> ENV:
> Spark:0.9.0-incubating
> Hadoop:2.3.0
>
> I run spark task on Yarn. I see the log in Nodemanager:
> 2014-09-25 17:43:34,141 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

> Memory usage of ProcessTree 549 for container-id 
> container_1411635522254_0001_01_000005: *4.5 GB**of 5 GB* physical 
> memory used; 5.0 GB of 10.5 GB virtual memory used
> 2014-09-25 17:43:37,171 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

> Memory usage of ProcessTree 549 for container-id 
> container_1411635522254_0001_01_000005: *4.5 GB**of 5 GB* physical 
> memory used; 5.0 GB of 10.5 GB virtual memory used
> 2014-09-25 17:43:40,210 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

> Memory usage of ProcessTree 549 for container-id 
> container_1411635522254_0001_01_000005: *4.5 GB**of 5 GB* physical 
> memory used; 5.0 GB of 10.5 GB virtual memory used
> 2014-09-25 17:43:43,239 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

> Memory usage of ProcessTree 549 for container-id 
> container_1411635522254_0001_01_000005: *4.5 GB of 5 GB* physical 
> memory used; 5.0 GB of 10.5 GB virtual memory used
> 2014-09-25 17:43:46,269 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:

> Memory usage of ProcessTree 549 for container-id 
> container_1411635522254_0001_01_000005: *4.5 GB of 5 GB* physical 
> memory used; 5.0 GB of 10.5 GB virtual memory used
>
> My task parameter is :
> --num-workers 4 --master-memory 2g --worker-memory 4g --worker-cores 4
> In myopinion "--worker-memory 4g" 4g is the maximum memory for 
> container .
> But why "4.5 GB of 5 GB physical memory used" in the log?
> And where to config "5G" maxinum memory for container?
>
> -- 
>
> WangXiaoyu
------------------------------------------------------------------------------------- 


Mime
View raw message