spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris DuBois <chris.dub...@gmail.com>
Subject Error: No space left on device
Date Wed, 16 Jul 2014 06:35:54 GMT
Hi all,

I am encountering the following error:

INFO scheduler.TaskSetManager: Loss was due to java.io.IOException: No
space left on device [duplicate 4]

For each slave, df -h looks roughtly like this, which makes the above error
surprising.

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  4.4G  3.5G  57% /
tmpfs                 7.4G  4.0K  7.4G   1% /dev/shm
/dev/xvdb              37G  3.3G   32G  10% /mnt
/dev/xvdf              37G  2.0G   34G   6% /mnt2
/dev/xvdv             500G   33M  500G   1% /vol

I'm on an EC2 cluster (c3.xlarge + 5 x m3) that I launched using the
spark-ec2 scripts and a clone of spark from today. The job I am running
closely resembles the collaborative filtering example
<https://spark.apache.org/docs/latest/mllib-collaborative-filtering.html>.
This issue happens with the 1M version as well as the 10 million rating
version of the MovieLens dataset.

I have seen previous
<http://mail-archives.apache.org/mod_mbox/spark-user/201403.mbox/%3C532F5AEC.8060805@nengoiksvelzud.com%3E>
 questions
<https://groups.google.com/forum/#!msg/spark-users/Axx4optAj-E/q5lWMv-ZqnwJ>,
but they haven't helped yet. For example, I tried setting the Spark tmp
directory to the EBS volume at /vol/, both by editing the spark conf file
(and copy-dir'ing it to the slaves) as well as through the SparkConf. Yet I
still get the above error. Here is my current Spark config below. Note that
I'm launching via ~/spark/bin/spark-submit.

conf = SparkConf()
conf.setAppName("RecommendALS").set("spark.local.dir",
"/vol/").set("spark.executor.memory", "7g").set("spark.akka.frameSize",
"100").setExecutorEnv("SPARK_JAVA_OPTS", " -Dspark.akka.frameSize=100")
sc = SparkContext(conf=conf)

Thanks for any advice,
Chris

Mime
View raw message