spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From condor join <spark_ker...@outlook.com>
Subject 答复: G1 GC takes too much time
Date Mon, 30 May 2016 02:17:11 GMT
The follwing are the parameters:
-XX:+UseG1GC
-XX:+UnlockDiagnostivVMOptions
-XX:G1SummarizeConcMark
-XX:InitiatingHeapOccupancyPercent=35
spark.executor.memory=4G

________________________________
发件人: Ted Yu <yuzhihong@gmail.com>
发送时间: 2016年5月30日 9:47:05
收件人: condor join
抄送: user@spark.apache.org
主题: Re: G1 GC takes too much time

bq. It happens during the Reduce majority.

Did the above refer to reduce operation ?

Can you share your G1GC parameters (and heap size for workers) ?

Thanks

On Sun, May 29, 2016 at 6:15 PM, condor join <spark_kernal@outlook.com<mailto:spark_kernal@outlook.com>>
wrote:
Hi,
my spark application failed due to take too much time during GC. Looking at the logs I found
these things:
1.there are Young GC takes too much time,and not found Full GC happen this;
2.The time takes too much during the object copy;
3.It happened  more easily when there were not enough resources;
4.It happens during the Reduce majority.

have anyone met the same question?
thanks



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>

Mime
View raw message