spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: Questions about the files that Spark will produce during its running
Date Tue, 29 Oct 2013 06:14:10 GMT
The error is from a worker node -- did you check that /data2 is set up properly on the worker
nodes too? In general that should be the only directory used.


On Oct 28, 2013, at 6:52 PM, Shangyu Luo <> wrote:

> Hello,
> I have some questions about the files that Spark will create and use during its running.
> (1) I am running a python program on Spark with a cluster of EC2. The data comes from
hdfs file system.  I have met the following error in the console of the master node:
> /data2/tmp/spark-local-20131029003412-c340/1b/shuffle_1_527_79
(No space left on device)
>         at Method)
>         at<init>(
>         at$
>         at$DiskBlockObjectWriter.write(DiskStore.scala:107)
>         at org.apache.spark.scheduler.ShuffleMapTask$$anonfun$run$1.apply(ShuffleMapTask.scala:152)
>         at org.apache.spark.scheduler.ShuffleMapTask$$anonfun$run$1.apply(ShuffleMapTask.scala:149)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>         at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
>         at
>         at
>         at org.apache.spark.executor.Executor$
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
> I set spark.local.dir=/data2/tmp in and there is about 800G space in data2
directory. I have checked the space of data2 and it is just used about 3G.
> So why Spark thinks that there is no space left on device?
> (2) Moreover, I am wondering if Spark will create some files  under other directories
other than spark.local.dir? Presently I use in some part
of my program, where will the persisted data be stored?
> (3) Lastly, I also had "Removing BlockManager xxx with no recent heart beats: xxxxxms
exceeds 45000ms " error sometimes. I have set the corresponding parameters in
> SPARK_JAVA_OPTS+="-Dspark.akka.timeout=300000 "
> SPARK_JAVA_OPTS+="-Dspark.worker.timeout=300000 "
> SPARK_JAVA_OPTS+="-Dspark.akka.askTimeout=3000 "
> SPARK_JAVA_OPTS+="-Dspark.akka.retry.wait=300000 "
> But it does no help. Can someone gives me some suggestion about solving this problem?
> Any help will be appreciated!
> Thanks!
> Best,
> Shangyu

View raw message