spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Tian <tianyi.asiai...@gmail.com>
Subject Re: Memory used in Spark-0.9.0-incubating
Date Thu, 25 Sep 2014 13:43:05 GMT
You should check the log of resource manager when you submit this job to yarn.

It will be recorded how many resources your spark application actually asked from resource
manager for each container.

Did you use fair scheduler?

there is a config parameter of fair scheduler “yarn.scheduler.increment-allocation-mb”,
default is 1024

it means if you ask 4097mb memory for a container, the resource manager will create a container
which use 5120mb memory.

But I can’t figure out where 5GB come from.

Maybe there are some codes which mistake 1024 and 1000?

Best Regards,

Yi Tian
tianyi.asiainfo@gmail.com




On Sep 25, 2014, at 18:41, 王晓雨 <wangxiaoyu1@jd.com> wrote:

> My yarn-site.xml config:
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>16384</value>
> </property>
> 
>> ENV:
>> Spark:0.9.0-incubating
>> Hadoop:2.3.0
>> 
>> I run spark task on Yarn. I see the log in Nodemanager:
>> 2014-09-25 17:43:34,141 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 549 for container-id container_1411635522254_0001_01_000005: 4.5
GB of 5 GB physical memory used; 5.0 GB of 10.5 GB virtual memory used
>> 2014-09-25 17:43:37,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 549 for container-id container_1411635522254_0001_01_000005: 4.5
GB of 5 GB physical memory used; 5.0 GB of 10.5 GB virtual memory used
>> 2014-09-25 17:43:40,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 549 for container-id container_1411635522254_0001_01_000005: 4.5
GB of 5 GB physical memory used; 5.0 GB of 10.5 GB virtual memory used
>> 2014-09-25 17:43:43,239 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 549 for container-id container_1411635522254_0001_01_000005: 4.5
GB of 5 GB physical memory used; 5.0 GB of 10.5 GB virtual memory used
>> 2014-09-25 17:43:46,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 549 for container-id container_1411635522254_0001_01_000005: 4.5
GB of 5 GB physical memory used; 5.0 GB of 10.5 GB virtual memory used
>> 
>> My task parameter is :
>> --num-workers 4 --master-memory 2g --worker-memory 4g --worker-cores 4
>> In my opinion "--worker-memory 4g" 4g is the maximum memory for container . 
>> But why "4.5 GB of 5 GB physical memory used" in the log?
>> And where to config "5G" maxinum memory for container?
>> 
>> -- 
>> 
>> WangXiaoyu
> -------------------------------------------------------------------------------------


Mime
View raw message