spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Wang <wh.s...@gmail.com>
Subject OutofMemory: Failed on spark/examples/bagel/WikipediaPageRank.scala
Date Mon, 19 May 2014 16:59:36 GMT
Hi, all

I am running a 30GB Wikipedia dataset on a 7-server cluster. Using
WikipediaPageRank underexample/Bagel.

My Spark version is bae07e3 [behind 1] fix different versions of
commons-lang dependency and apache/spark#746 addendum

The problem is that the job will fail after several stages because of
OutofMemory
Error. The reason might be that the default executor's memory size is *512M*
.

I try to modify executor memory size via export
SPARK_JAVA_OPTS="-Dspark-cores-max=8 -Dspark.executor.memory=8g", but
SPARK_JAVA_OPTS is not recommended in Spark 1.0+. Log also tells ERROR
SparkConf.

   - Anyone knows the difference between executor memory / cores and worker
   memory /cores?
   - How to set the executor memory in Spark 1.0+?

spark-env.sh:

export SPARK_WORKER_MEMORY=2g
export SPARK_MASTER_IP=192.168.1.12
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=2
export SPARK_WORKER_INSTANCES=2

Each server has 8G men and 8-core CPU. But after several stages, the job
failed and outputs following logs:

14/05/19 22:29:32 WARN TaskSetManager: Loss was due to
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
14/05/19 22:29:32 INFO SparkDeploySchedulerBackend: Executor 10
disconnected, so removing it
14/05/19 22:29:32 ERROR TaskSchedulerImpl: Lost executor 10 on
host125: remote Akka client disassociat
...
14/05/19 22:29:33 ERROR SparkDeploySchedulerBackend: Application has
been killed. Reason: Master removed our application: FAILED
14/05/19 22:29:33 WARN TaskSetManager: Loss was due to fetch failure
from BlockManagerId(10, host125,
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:735)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    ...
14/05/19 22:29:33 INFO DAGScheduler: Failed to run foreach at Bagel.scala:251
Exception in thread "main" org.apache.spark.SparkException: Job
aborted due to stage failure: Master removed our application: FAILED
14/05/19 22:29:33 INFO TaskSchedulerImpl: Cancelling stage 4
14/05/19 22:29:33 INFO TaskSchedulerImpl: Stage 4 was cancelled
14/05/19 22:29:33 WARN TaskSetManager: Loss was due to java.io.IOException
java.io.IOException: Failed on local exception:
java.io.InterruptedIOException: Interruped while waiting for IO on
channel java.nio.chan
nels.SocketChannel[connected local=/192.168.1.123:54254
remote=/192.168.1.12:9000]. 59922 millis timeout left.; Host Details :
local hos
t is: "host123/192.168.1.123"; destination host is: "sing12":9000;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
    ...


Regards,
Wang Hao(王灏)

CloudTeam | School of Software Engineering
Shanghai Jiao Tong University
Address:800 Dongchuan Road, Minhang District, Shanghai, 200240
Email:wh.sjtu@gmail.com

Mime
View raw message