spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Yoon <sungji...@gmail.com>
Subject OutOfMemorryError after chaning to .persist(StorageLevel.MEMORY_ONLY_SER)
Date Thu, 03 Mar 2016 11:48:36 GMT
Hi, Spark users.

I am getting the following OutOfMemoryError: Java heap space after changing
to StorageLevel.MEMORY_ONLY_SER.
MEMORY_AND_DISK_SER also throws the same error.
I thought DISK option should put unfitting blocks to the disk.
What could cause the OOM in such situation?

Is there any good approach to solve this issue?

FYI, I am using 1.6.0 with stand-alone mode.
Single node with master and slave in the same node.
16 Cores and 64GB mem.


16/03/03 11:09:27 WARN BlockManager: Putting block rdd_39_0 failed
16/03/03 11:09:27 ERROR Executor: Exception in task 0.0 in stage 17.0 (TID 37)
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
	at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
	at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1196)
	at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202)
	at org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:136)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:800)
	at org.apache.spark.storage.BlockManager.putArray(BlockManager.scala:676)
	at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:175)
	at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/03/03 11:09:27 ERROR SparkUncaughtExceptionHandler: Uncaught
exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
	at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
	at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1196)
	at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202)
	at org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:136)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:800)
	at org.apache.spark.storage.BlockManager.putArray(BlockManager.scala:676)
	at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:175)
	at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Mime
View raw message