spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh <josh.erick...@youneeq.ca>
Subject Deleting temp dir Exception
Date Mon, 03 Nov 2014 23:08:58 GMT
Hi,

I've written a short scala app to perform word counts on a text file and am
getting the following exception as the program completes (after it prints
out all of the word counts).

Exception in thread "delete Spark temp dir
C:\Users\Josh\AppData\Local\Temp\spark-0fdd0b79-7329-4690-a093-0fdb0d21e32c"
java.io.IOException: Failed to delete:
C:\Users\Josh\AppData\Local\Temp\spark-0fdd0b79-7329-4690-a093-0fdb0d21e32c\word-count_2.10-1.0.jar
        at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:692)
        at
org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:686)
        at
org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:685)
        at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at
scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
        at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:685)
        at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:281)

A few details: Spark version 1.1.0 built from source on Windows 8. Scala
version 2.11.4, JRE version 7 Update 71. Everything is local on a single
machine. I can run the code in the scala shell just fine, this appears to be
an issue with not being able to delete the temporary JAR file after running
the application via spark-submit. I've checked the location of the JAR file
and it is indeed left behind, so it is not being deleted. I guess my
question is why is it unable to be deleted and is there anything I can do
differently to fix this?

I do not see this exception when I run the SimpleApp example taken from
here:
http://spark.apache.org/docs/latest/quick-start.html#standalone-applications

Here is the code for my WordCount app that is producing this exception:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object WordCount {
	def main(args: Array[String]) {
		val filename = "data/rideshare.txt"
		val conf = new SparkConf().setAppName("Word Count").setMaster("local")
    	val sc = new SparkContext(conf)
		val file = sc.textFile(filename, 2).cache()
		val wordCounts = file.flatMap(line => line.split(" ")).map(word => (word,
1)).reduceByKey((a, b) => a + b)
		wordCounts.collect().foreach(println)
	}
}

Thanks,
Josh



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deleting-temp-dir-Exception-tp18006.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message