spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StanZhai <m...@zhaishidan.cn>
Subject Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0
Date Fri, 03 Feb 2017 03:40:28 GMT
CentOS 7.1,
Linux version 3.10.0-229.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc
version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42
UTC 2015


Michael Allman-2 wrote
> Hi Stan,
> 
> What OS/version are you using?
> 
> Michael
> 
>> On Jan 22, 2017, at 11:36 PM, StanZhai &lt;

> mail@

> &gt; wrote:
>> 
>> I'm using Parallel GC.
>> rxin wrote
>>> Are you using G1 GC? G1 sometimes uses a lot more memory than the size
>>> allocated.
>>> 
>>> 
>>> On Sun, Jan 22, 2017 at 12:58 AM StanZhai &lt;
>> 
>>> mail@
>> 
>>> &gt; wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> 
>>>> 
>>>> We just upgraded our Spark from 1.6.2 to 2.1.0.
>>>> 
>>>> 
>>>> 
>>>> Our Spark application is started by spark-submit with config of
>>>> 
>>>> `--executor-memory 35G` in standalone model, but the actual use of
>>>> memory
>>>> up
>>>> 
>>>> to 65G after a full gc(jmap -histo:live $pid) as follow:
>>>> 
>>>> 
>>>> 
>>>> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>>>> 
>>>> test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21
>>>> 
>>>> /home/test/service/jdk/bin/java -cp
>>>> 
>>>> 
>>>> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>>>> 
>>>> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>>>> 
>>>> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>>>> 
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>>>> 
>>>> spark://
>> 
>>> CoarseGrainedScheduler@.xxx
>> 
>>> :47781 --executor-id 1
>>>> 
>>>> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
>>>> --worker-url
>>>> 
>>>> spark://Worker@test-192:33890
>>>> 
>>>> 
>>>> 
>>>> Our Spark jobs are all sql.
>>>> 
>>>> 
>>>> 
>>>> The exceed memory looks like off-heap memory, but the default value of
>>>> 
>>>> `spark.memory.offHeap.enabled` is `false`.
>>>> 
>>>> 
>>>> 
>>>> We didn't find the problem in Spark 1.6.x, what causes this in Spark
>>>> 2.1.0?
>>>> 
>>>> 
>>>> 
>>>> Any help is greatly appreicated!
>>>> 
>>>> 
>>>> 
>>>> Best,
>>>> 
>>>> Stan
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> View this message in context:
>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>>>> 
>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>> Nabble.com &lt;http://nabble.com/&gt;.
>>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> 
>>>> To unsubscribe e-mail: 
>> 
>>> dev-unsubscribe@.apache
>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html
>> &lt;http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html&gt;
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com &lt;http://nabble.com/&gt;.
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: 

> dev-unsubscribe@.apache

>  &lt;mailto:

> dev-unsubscribe@.apache

> &gt;





--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20833.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message