spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈韵竹 <anny9...@gmail.com>
Subject Re: still "GC overhead limit exceeded" after increasing heap space
Date Wed, 01 Oct 2014 21:21:42 GMT
Thanks Sean. This is how I set this memory. I set it when I start to run
the job

java -Xms64g -Xmx64g -cp
/root/spark/lib/spark-assembly-1.0.0-hadoop1.0.4.jar:/root/scala/lib/scala-library.jar:./target/MyProject.jar
MyClass

Is there some problem with it?



On Wed, Oct 1, 2014 at 2:03 PM, Sean Owen <sowen@cloudera.com> wrote:

> How are you setting this memory? You may be configuring the wrong
> process's memory, like the driver and not the executors.
> On Oct 1, 2014 9:37 PM, "anny9699" <anny9699@gmail.com> wrote:
>
>> Hi,
>>
>> After reading some previous posts about this issue, I have increased the
>> java heap space to "-Xms64g -Xmx64g", but still met the
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. Does
>> anyone
>> have other suggestions?
>>
>> I am reading a data of 200 GB and my total memory is 120 GB, so I use
>> "MEMORY_AND_DISK_SER" and kryo serialization.
>>
>> Thanks a lot!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/still-GC-overhead-limit-exceeded-after-increasing-heap-space-tp15540.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>

Mime
View raw message