spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: 答复: G1 GC takes too much time
Date Mon, 30 May 2016 03:39:40 GMT
Please consider reading G1GC tuning guide(s).
Here is an example:

http://product.hubspot.com/blog/g1gc-tuning-your-hbase-cluster

On Sun, May 29, 2016 at 7:17 PM, condor join <spark_kernal@outlook.com>
wrote:

> The follwing are the parameters:
> -XX:+UseG1GC
> -XX:+UnlockDiagnostivVMOptions
> -XX:G1SummarizeConcMark
> -XX:InitiatingHeapOccupancyPercent=35
> spark.executor.memory=4G
>
> ------------------------------
> *发件人:* Ted Yu <yuzhihong@gmail.com>
> *发送时间:* 2016年5月30日 9:47:05
> *收件人:* condor join
> *抄送:* user@spark.apache.org
> *主题:* Re: G1 GC takes too much time
>
> bq. It happens during the Reduce majority.
>
> Did the above refer to reduce operation ?
>
> Can you share your G1GC parameters (and heap size for workers) ?
>
> Thanks
>
> On Sun, May 29, 2016 at 6:15 PM, condor join <spark_kernal@outlook.com>
> wrote:
>
>> Hi,
>> my spark application failed due to take too much time during GC. Looking
>> at the logs I found these things:
>> 1.there are Young GC takes too much time,and not found Full GC happen
>> this;
>> 2.The time takes too much during the object copy;
>> 3.It happened  more easily when there were not enough resources;
>> 4.It happens during the Reduce majority.
>>
>> have anyone met the same question?
>> thanks
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>

Mime
View raw message