spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammed Guller <moham...@glassbeam.com>
Subject RE: unknown issue in submitting a spark job
Date Fri, 30 Jan 2015 00:32:57 GMT
How much memory are you assigning to the Spark executor on the worker node?

Mohammed

From: ey-chih chow [mailto:eychih@hotmail.com]
Sent: Thursday, January 29, 2015 3:35 PM
To: Mohammed Guller; user@spark.apache.org
Subject: RE: unknown issue in submitting a spark job

The worker node has 15G memory, 1x32 GB SSD, and 2 core.  The data file is from S3. If I don't
set mapred.max.split.size, it is fine with only one partition.  Otherwise, it will generate
OOME.

Ey-Chih Chow

> From: mohammed@glassbeam.com<mailto:mohammed@glassbeam.com>
> To: eychih@hotmail.com<mailto:eychih@hotmail.com>; user@spark.apache.org<mailto:user@spark.apache.org>
> Subject: RE: unknown issue in submitting a spark job
> Date: Thu, 29 Jan 2015 21:16:13 +0000
>
> Looks like the application is using a lot more memory than available. Could be a bug
somewhere in the code or just underpowered machine. Hard to say without looking at the code.
>
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Mohammed
>
>
> -----Original Message-----
> From: ey-chih chow [mailto:eychih@hotmail.com]
> Sent: Thursday, January 29, 2015 1:06 AM
> To: user@spark.apache.org<mailto:user@spark.apache.org>
> Subject: unknown issue in submitting a spark job
>
> Hi,
>
> I submitted a job using spark-submit and got the following exception.
> Anybody knows how to fix this? Thanks.
>
> Ey-Chih Chow
>
> ============================================
>
> 15/01/29 08:53:10 INFO storage.BlockManagerMasterActor: Registering block manager ip-10-10-8-191.us-west-2.compute.internal:47722
with 6.6 GB RAM Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:265)
> at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:94)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1128)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:935)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:109)
> at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> ... 6 more
> 15/01/29 08:54:33 INFO storage.BlockManager: Removing RDD 1
> 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: exception on LARS' timer thread
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397)
> at akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
> at java.lang.Thread.run(Thread.java:745)
> 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-scheduler-1]
shutting down ActorSystem [sparkDriver]
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397)
> at akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
> at java.lang.Thread.run(Thread.java:745)
> 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: exception on LARS' timer thread
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at akka.dispatch.AbstractNodeQueue.<init>(AbstractNodeQueue.java:19)
> at
> akka.actor.LightArrayRevolverScheduler$TaskQueue.<init>(Scheduler.scala:431)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397)
> at akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
> at java.lang.Thread.run(Thread.java:745)
> 15/01/29 08:54:33 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [Driver-scheduler-1]
shutting down ActorSystem [Driver]
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at akka.dispatch.AbstractNodeQueue.<init>(AbstractNodeQueue.java:19)
> at
> akka.actor.LightArrayRevolverScheduler$TaskQueue.<init>(Scheduler.scala:431)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:397)
> at akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
> at java.lang.Thread.run(Thread.java:745)
> 15/01/29 08:54:33 WARN storage.BlockManagerMasterActor: Removing BlockManager BlockManagerId(0,
ip-10-10-8-191.us-west-2.compute.internal,
> 47722, 0) with no recent heart beats: 82575ms exceeds 45000ms
> 15/01/29 08:54:33 INFO spark.ContextCleaner: Cleaned RDD 1
> 15/01/29 08:54:33 WARN util.AkkaUtils: Error sending message in 1 attempts
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/BlockManagerMaster#-538003375]] had already been
terminated.
> at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
> at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:175)
> at
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:218)
> at
> org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:126)
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/unknown-issue-in-submitting-a-spark-job-tp21418.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>
>

Mime
View raw message