spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yong Zhang <>
Subject Spark 1.5.2 Master OOM
Date Wed, 30 Mar 2016 15:46:20 GMT
Hi, Sparkers
Our cluster is running Spark 1.5.2 with Standalone mode.
It runs fine for weeks, but today, I found out the master crash due to OOM.
We have several ETL jobs runs daily on Spark, and adhoc jobs. I can see the "Completed Applications"
table grows in the master UI.
Original I set "export SPARK_DAEMON_MEMORY=1g", as I don't think master/worker JVM daemon
needs too much memory.
I never meet Spar master OOM when we run on version 1.3.1. But suddenly, I got it on 1.5.2.
I am not sure if it is due to growing "Completed Applications" history? I didn't start the
spark history server, as we don't have that requirement yet.
Now I change the daemon memory from 1g to 2g, and restart the cluster.
Below is the OOM log in the spark master. In fact, OOM happened minutes after a job just finished.
The job finished successfully, as the final HDFS output was generated around 09:31:00. So
in theory, there is no active jobs while OOM happens, or it is trigged by the succeeds of
that job. I don't know. But Spark master OOM in fact is a SPOF for us. Does anyone have any
idea about it?
16/03/30 09:36:40 ERROR akka.ErrorMonitor: Uncaught fatal error from thread [sparkMaster-akka.remote.default-remote-dispatcher-33]
shutting down ActorSystem [sparkMaster]java.lang.OutOfMemoryError: Java heap space       
at java.lang.Class.getDeclaredMethods0(Native Method)        at java.lang.Class.privateGetDeclaredMethods(
       at java.lang.Class.getDeclaredMethod(        at
       at$1700(        at$
       at$        at
Method)        at<init>(     
  at        at
       at        at        at
       at        at
       at        at
       at        at        at
at        at
       at        at
       at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)       
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)        at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
       at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
       at scala.util.Try$.apply(Try.scala:161)        at akka.serialization.Serialization.deserialize(Serialization.scala:98)
View raw message