That will not work, I am currently in the same dilemma. You need to pass some parameters to the JVMs running on the slaves like Xms and Xmx. I managed to do that while running the workload locally but I could not do this on the cluster.
Changing SPARK_DAEMON_MEMORY will have no effect, the memory used will be 512 M


On Thursday, January 23, 2014 11:23 PM, Manoj Samel <manojsameltech@gmail.com> wrote:
I could increase the memory for master.

However, my understanding of the Master is that it just does the DAG scheduling for workers and does not do any RDD processing itself. If this is true and since only one application was running; does the master needs > 512 Mb just to execute the DAG for 3105 tasks ?


On Thu, Jan 23, 2014 at 11:52 AM, Guillaume Pitel <guillaume.pitel@exensa.com> wrote:
Why not trying to increase the master memory ?

export SPARK_DAEMON_MEMORY=1g in master spark-env.sh

Guillaume
I have a standalone master cluster with 512Mb for master and 12 g for each worker. I start spark-shell with 10g and spark.task.maxFailures=99999

I run a job that plans 3105 tasks. 3104 tasks out of 3105 tasks run OK, then the tasks start failing and after 36K failed tasks, I get following. Is the master running out of memory ?

Exception in thread "DAGScheduler" java.lang.OutOfMemoryError: GC overhead limit exceeded
Jan 232014 6:33:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught error from thread [spark-11] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jan 232014 6:36:43 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jan 232014 6:36:36 PM org.jboss.netty.channel.socket.nio.AbstractNioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded






--
eXenSa
Guillaume PITEL, Président
+33(0)6 25 48 86 80 / +33(0)9 70 44 67 53

eXenSa S.A.S.
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05