spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guru Medasani <gdm...@outlook.com>
Subject Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Date Wed, 28 Jan 2015 16:46:09 GMT
Hi Antony, Did you get pass this error by repartitioning your job with 
smaller tasks as Sven Krasser pointed out?

From:  Antony Mayi <antonymayi@yahoo.com>
Reply-To:  Antony Mayi <antonymayi@yahoo.com>
Date:  Tuesday, January 27, 2015 at 5:24 PM
To:  Guru Medasani <gdmeda@outlook.com>, Sven Krasser <krasser@gmail.com>
Cc:  Sandy Ryza <sandy.ryza@cloudera.com>, "user@spark.apache.org" 
<user@spark.apache.org>
Subject:  Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

I have yarn configured with yarn.nodemanager.vmem-check-enabled=false and 
yarn.nodemanager.pmem-check-enabled=false to avoid yarn killing the 
containers.

the stack trace is bellow.

thanks,
Antony.

15/01/27 17:02:53 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED 
SIGNAL 15: SIGTERM
15/01/27 17:02:53 ERROR executor.Executor: Exception in task 21.0 in stage 
12.0 (TID 1312)
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Integer.valueOf(Integer.java:642)
        at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:70)
        at 
scala.collection.mutable.ArrayOps$ofInt.apply(ArrayOps.scala:156)
        at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scal
a:33)
        at 
scala.collection.mutable.ArrayOps$ofInt.foreach(ArrayOps.scala:156)
        at scala.collection.SeqLike$class.distinct(SeqLike.scala:493)
        at 
scala.collection.mutable.ArrayOps$ofInt.distinct(ArrayOps.scala:156)
        at 
org.apache.spark.mllib.recommendation.ALS.org$apache$spark$mllib$recommenda
tion$ALS$$makeOutLinkBlock(ALS.scala:404)
        at 
org.apache.spark.mllib.recommendation.ALS$$anonfun$15.apply(ALS.scala:459)
        at 
org.apache.spark.mllib.recommendation.ALS$$anonfun$15.apply(ALS.scala:456)
        at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:614)
        at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:614)
        at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
        at 
org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at 
org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:61)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:228)
        at 
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.sca
la:130)
        at 
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.sca
la:127)
        at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(Traver
sableLike.scala:772)
        at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scal
a:33)
        at 
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
        at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:7
71)
        at 
org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
        at 
org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
        at 
org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:
31)
15/01/27 17:02:53 ERROR util.SparkUncaughtExceptionHandler: Uncaught 
exception in thread Thread[Executor task launch worker-8,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Integer.valueOf(Integer.java:642)
        at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:70)
        at 
scala.collection.mutable.ArrayOps$ofInt.apply(ArrayOps.scala:156)
        at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scal
a:33)
        at 
scala.collection.mutable.ArrayOps$ofInt.foreach(ArrayOps.scala:156)
        at scala.collection.SeqLike$class.distinct(SeqLike.scala:493)
        at 
scala.collection.mutable.ArrayOps$ofInt.distinct(ArrayOps.scala:156)
        at 
org.apache.spark.mllib.recommendation.ALS.org$apache$spark$mllib$recommenda
tion$ALS$$makeOutLinkBlock(ALS.scala:404)
        at 
org.apache.spark.mllib.recommendation.ALS$$anonfun$15.apply(ALS.scala:459)
        at 
org.apache.spark.mllib.recommendation.ALS$$anonfun$15.apply(ALS.scala:456)
        at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:614)
        at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:614)
        at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
        at 
org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at 
org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:61)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:228)
        at 
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.sca
la:130)
        at 
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.sca
la:127)
        at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(Traver
sableLike.scala:772)
        at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scal
a:33)
        at 
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
        at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:7
71)
        at 
org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
        at 
org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
        at 
org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:
31)

 


 
 
 
  On Wednesday, 28 January 2015, 0:01, Guru Medasani <gdmeda@outlook.com> 
wrote:
  
 
  

 
Can you attach the logs where this is failing?

From:  Sven Krasser <krasser@gmail.com>
Date:  Tuesday, January 27, 2015 at 4:50 PM
To:  Guru Medasani <gdmeda@outlook.com>
Cc:  Sandy Ryza <sandy.ryza@cloudera.com>, Antony Mayi 
<antonymayi@yahoo.com>, "user@spark.apache.org" <user@spark.apache.org>
Subject:  Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Since it's an executor running OOM it doesn't look like a container being 
killed by YARN to me. As a starting point, can you repartition your job 
into smaller tasks?
-Sven

On Tue, Jan 27, 2015 at 2:34 PM, Guru Medasani <gdmeda@outlook.com> wrote:
Hi Anthony,

What is the setting of the total amount of memory in MB that can be 
allocated to containers on your NodeManagers?

yarn.nodemanager.resource.memory-mb

Can you check this above configuration in yarn-site.xml used by the node 
manager process?

-Guru Medasani

From:  Sandy Ryza <sandy.ryza@cloudera.com>
Date:  Tuesday, January 27, 2015 at 3:33 PM
To:  Antony Mayi <antonymayi@yahoo.com>
Cc:  "user@spark.apache.org" <user@spark.apache.org>
Subject:  Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi Antony,

If you look in the YARN NodeManager logs, do you see that it's killing the 
executors?  Or are they crashing for a different reason?

-Sandy

On Tue, Jan 27, 2015 at 12:43 PM, Antony Mayi 
<antonymayi@yahoo.com.invalid> wrote:
Hi,

I am using spark.yarn.executor.memoryOverhead=8192 yet getting executors 
crashed with this error.

does that mean I have genuinely not enough RAM or is this matter of config 
tuning?

other config options used:
spark.storage.memoryFraction=0.3
SPARK_EXECUTOR_MEMORY=14G

running spark 1.2.0 as yarn-client on cluster of 10 nodes (the workload is 
ALS trainImplicit on ~15GB dataset)

thanks for any ideas,
Antony.




-- 
http://sites.google.com/site/krasser/?utm_source=sig


 
  
 
   
 


Mime
View raw message