spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: No space left on device exception
Date Sun, 23 Mar 2014 22:35:05 GMT
On some systems, /tmp/ is an in-memory tmpfs file system, with its own size
limit. It's possible that this limit has been exceeded. You might try
running the "df" command to check to free space of "/tmp" or root if tmp
isn't listed.

3 GB also seems pretty low for the remaining free space of a disk. If your
disk size is in the TB range, it's possible that the last couple GB have
issues when being allocated due to fragmentation or reclamation policies.


On Sun, Mar 23, 2014 at 3:06 PM, Ognen Duzlevski
<ognen@nengoiksvelzud.com>wrote:

> Hello,
>
> I have a weird error showing up when I run a job on my Spark cluster. The
> version of spark is 0.9 and I have 3+ GB free on the disk when this error
> shows up. Any ideas what I should be looking for?
>
> [error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
> 167.0:3 failed 4 times (most recent failure: Exception failure:
> java.io.FileNotFoundException: /tmp/spark-local-20140323214638-72df/31/shuffle_31_3_127
> (No space left on device))
> org.apache.spark.SparkException: Job aborted: Task 167.0:3 failed 4 times
> (most recent failure: Exception failure: java.io.FileNotFoundException:
> /tmp/spark-local-20140323214638-72df/31/shuffle_31_3_127 (No space left
> on device))
>     at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$
> apache$spark$scheduler$DAGScheduler$$abortStage$1.
> apply(DAGScheduler.scala:1028)
>     at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$
> apache$spark$scheduler$DAGScheduler$$abortStage$1.
> apply(DAGScheduler.scala:1026)
>     at scala.collection.mutable.ResizableArray$class.foreach(
> ResizableArray.scala:59)
>     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>     at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$
> scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
>     at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> processEvent$10.apply(DAGScheduler.scala:619)
>     at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> processEvent$10.apply(DAGScheduler.scala:619)
>     at scala.Option.foreach(Option.scala:236)
>     at org.apache.spark.scheduler.DAGScheduler.processEvent(
> DAGScheduler.scala:619)
>     at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$
> $anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
>
> Thanks!
> Ognen
>

Mime
View raw message