spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kal El <pinu.datri...@yahoo.com>
Subject Re: Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC overhead limit exceeded
Date Thu, 23 Jan 2014 21:27:29 GMT
That will not work, I am currently in the same dilemma. You need to pass some parameters to
the JVMs running on the slaves like Xms and Xmx. I managed to do that while running the workload
locally but I could not do this on the cluster.
Changing SPARK_DAEMON_MEMORY will have no effect, the memory used will be 512 M



On Thursday, January 23, 2014 11:23 PM, Manoj Samel <manojsameltech@gmail.com> wrote:
 
I could increase the memory for master.

However, my understanding of the Master is that it just does the DAG scheduling for workers
and does not do any RDD processing itself. If this is true and since only one application
was running; does the master needs > 512 Mb just to execute the DAG for 3105 tasks ?



On Thu, Jan 23, 2014 at 11:52 AM, Guillaume Pitel <guillaume.pitel@exensa.com> wrote:

Why not trying to increase the master memory ?
>
>export SPARK_DAEMON_MEMORY=1g in master spark-env.sh
>
>Guillaume
>
>I have a standalone master cluster with 512Mb for master and 12 g for each worker. I start
spark-shell with 10g and spark.task.maxFailures=99999
>>
>>I run a job that plans 3105 tasks. 3104 tasks out of 3105 tasks run OK, then the tasks
start failing and after 36K failed tasks, I get following. Is the master running out of memory
?
>>
>>
>>Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC overhead limit
exceeded
>>Jan 23, 2014 6:33:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>WARNING: Unexpected exception in the selector loop.
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Uncaught error from thread [spark-11] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is
enabled for ActorSystem[spark]
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Jan 23, 2014 6:36:43 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>WARNING: Unexpected exception in the selector loop.
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>Jan 23, 2014 6:36:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>WARNING: Unexpected exception in the selector loop.
>>java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>-- 
>
> Guillaume PITEL, Président 
>+33(0)6 25 48 86 80 / +33(0)9 70 44 67 53
>  
>eXenSa S.A.S.  
>41, rue Périer - 92120 Montrouge - FRANCE 
>Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05    
Mime
View raw message