spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henggang Cui <cuihengg...@gmail.com>
Subject Setting spark memory limit
Date Mon, 09 Jun 2014 22:24:27 GMT
Hi,

I'm trying to run the SimpleApp example (
http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala)
on a larger dataset.

The input file is about 1GB, but when I run the Spark program, it
says:"java.lang.OutOfMemoryError: GC overhead limit exceeded", the full
error output is attached at the end of the E-mail.

Then I tried multiple ways of setting the memory limit.

In SimpleApp.scala file, I set the following configurations:
    val conf = new SparkConf()
               .setAppName("Simple Application")
               .set("spark.executor.memory", "10g")

And I have also tried appending the following configuration to
conf/spark-defaults.conf file:
    spark.executor.memory   10g

But neither of them works. In the error message, it claims "(estimated size
103.8 MB, free 191.1 MB)", so the total available memory is still 300MB.
Why?

Thanks,
Cui


$ ~/spark-1.0.0-bin-hadoop1/bin/spark-submit --class "SimpleApp" --master
local[4] target/scala-2.10/simple-project_2.10-1.0.jar /tmp/mdata0-10.tsd
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
14/06/09 15:06:29 INFO SecurityManager: Using Spark's default log4j
profile: org/apache/spark/log4j-defaults.properties
14/06/09 15:06:29 INFO SecurityManager: Changing view acls to: cuihe
14/06/09 15:06:29 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(cuihe)
14/06/09 15:06:29 INFO Slf4jLogger: Slf4jLogger started
14/06/09 15:06:29 INFO Remoting: Starting remoting
14/06/09 15:06:30 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://spark@131-1.bfc.hpl.hp.com:40779]
14/06/09 15:06:30 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://spark@131-1.bfc.hpl.hp.com:40779]
14/06/09 15:06:30 INFO SparkEnv: Registering MapOutputTracker
14/06/09 15:06:30 INFO SparkEnv: Registering BlockManagerMaster
14/06/09 15:06:30 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20140609150630-eaa9
14/06/09 15:06:30 INFO MemoryStore: MemoryStore started with capacity 294.9
MB.
14/06/09 15:06:30 INFO ConnectionManager: Bound socket to port 47164 with
id = ConnectionManagerId(131-1.bfc.hpl.hp.com,47164)
14/06/09 15:06:30 INFO BlockManagerMaster: Trying to register BlockManager
14/06/09 15:06:30 INFO BlockManagerInfo: Registering block manager
131-1.bfc.hpl.hp.com:47164 with 294.9 MB RAM
14/06/09 15:06:30 INFO BlockManagerMaster: Registered BlockManager
14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server
14/06/09 15:06:30 INFO HttpBroadcast: Broadcast server started at
http://16.106.36.131:48587
14/06/09 15:06:30 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-35e1c47b-bfa1-4fba-bc64-df8eee287bb7
14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server
14/06/09 15:06:30 INFO SparkUI: Started SparkUI at
http://131-1.bfc.hpl.hp.com:4040
14/06/09 15:06:30 INFO SparkContext: Added JAR
file:/data/cuihe/spark-app/target/scala-2.10/simple-project_2.10-1.0.jar at
http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp
1402351590741
14/06/09 15:06:30 INFO MemoryStore: ensureFreeSpace(32856) called with
curMem=0, maxMem=309225062
14/06/09 15:06:30 INFO MemoryStore: Block broadcast_0 stored as values to
memory (estimated size 32.1 KB, free 294.9 MB)
14/06/09 15:06:30 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/09 15:06:30 WARN LoadSnappy: Snappy native library not loaded
14/06/09 15:06:30 INFO FileInputFormat: Total input paths to process : 1
14/06/09 15:06:30 INFO SparkContext: Starting job: count at
SimpleApp.scala:14
14/06/09 15:06:31 INFO DAGScheduler: Got job 0 (count at
SimpleApp.scala:14) with 7 output partitions (allowLocal=false)
14/06/09 15:06:31 INFO DAGScheduler: Final stage: Stage 0(count at
SimpleApp.scala:14)
14/06/09 15:06:31 INFO DAGScheduler: Parents of final stage: List()
14/06/09 15:06:31 INFO DAGScheduler: Missing parents: List()
14/06/09 15:06:31 INFO DAGScheduler: Submitting Stage 0 (FilteredRDD[2] at
filter at SimpleApp.scala:14), which has no missing parents
14/06/09 15:06:31 INFO DAGScheduler: Submitting 7 missing tasks from Stage
0 (FilteredRDD[2] at filter at SimpleApp.scala:14)
14/06/09 15:06:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 7 tasks
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on
executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:0 as 1839 bytes
in 2 ms
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on
executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:1 as 1839 bytes
in 0 ms
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:2 as TID 2 on
executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:2 as 1839 bytes
in 1 ms
14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:3 as TID 3 on
executor localhost: localhost (PROCESS_LOCAL)
14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:3 as 1839 bytes
in 1 ms
14/06/09 15:06:31 INFO Executor: Running task ID 0
14/06/09 15:06:31 INFO Executor: Running task ID 1
14/06/09 15:06:31 INFO Executor: Running task ID 2
14/06/09 15:06:31 INFO Executor: Running task ID 3
14/06/09 15:06:31 INFO Executor: Fetching
http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp
1402351590741
14/06/09 15:06:31 INFO Utils: Fetching
http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar to
/tmp/fetchFileTemp7241193225836706654.tmp
14/06/09 15:06:31 INFO Executor: Adding
file:/tmp/spark-68aa13c8-8146-4e6a-80a1-c406a4cef89f/simple-project_2.10-1.0.jar
to class loader
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_2 not found, computing
it
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_0 not found, computing
it
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_1 not found, computing
it
14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_3 not found, computing
it
14/06/09 15:06:31 INFO HadoopRDD: Input split:
file:/tmp/mdata0-10.tsd:67108864+33554432
14/06/09 15:06:31 INFO HadoopRDD: Input split:
file:/tmp/mdata0-10.tsd:33554432+33554432
14/06/09 15:06:31 INFO HadoopRDD: Input split:
file:/tmp/mdata0-10.tsd:100663296+33554432
14/06/09 15:06:31 INFO HadoopRDD: Input split:
file:/tmp/mdata0-10.tsd:0+33554432
14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108800293) called with
curMem=32856, maxMem=309225062
14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_2 stored as values to
memory (estimated size 103.8 MB, free 191.1 MB)
14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108716407) called with
curMem=108833149, maxMem=309225062
14/06/09 15:06:50 ERROR Executor: Exception in task ID 1
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
        at java.nio.CharBuffer.toString(CharBuffer.java:1201)
        at org.apache.hadoop.io.Text.decode(Text.java:350)
        at org.apache.hadoop.io.Text.decode(Text.java:327)
        at org.apache.hadoop.io.Text.toString(Text.java:254)
        at
org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at
org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at
org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
        at org.apache.spark.scheduler.Task.run(Task.scala:51)
        at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_2 in memory on
131-1.bfc.hpl.hp.com:47164 (size: 103.8 MB, free: 191.1 MB)
14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_3 stored as values to
memory (estimated size 103.7 MB, free 87.4 MB)
14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_2
14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_3 in memory on
131-1.bfc.hpl.hp.com:47164 (size: 103.7 MB, free: 87.5 MB)
14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_3
14/06/09 15:06:50 ERROR ExecutorUncaughtExceptionHandler: Uncaught
exception in thread Thread[Executor task launch worker-1,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
        at java.nio.CharBuffer.toString(CharBuffer.java:1201)
        at org.apache.hadoop.io.Text.decode(Text.java:350)
        at org.apache.hadoop.io.Text.decode(Text.java:327)
        at org.apache.hadoop.io.Text.toString(Text.java:254)
        at
org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at
org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at
org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
        at org.apache.spark.scheduler.Task.run(Task.scala:51)
        at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
14/06/09 15:06:50 INFO TaskSetManager: Starting task 0.0:4 as TID 4 on
executor localhost: localhost (PROCESS_LOCAL)

Mime
View raw message