Does the sparkContext shuts down itself by default even if I dont mention specifically in my code?? Because, I ran the application without sc.context(), still I get file system closed error along with correct output.

On Tue, Dec 2, 2014 at 2:20 PM, Akhil Das <akhil@sigmoidanalytics.com> wrote:
It could be because those threads are finishing quickly.

Thanks
Best Regards

On Tue, Dec 2, 2014 at 2:19 PM, rapelly kartheek <kartheek.mbms@gmail.com> wrote:
But, somehow, if I run this application for the second time, I find that the application gets executed and the results are out regardless of the same errors in logs.

On Tue, Dec 2, 2014 at 2:08 PM, Akhil Das <akhil@sigmoidanalytics.com> wrote:
Your code seems to have a lot of threads and i think you might be invoking sc.stop before those threads get finished.

Thanks
Best Regards

On Tue, Dec 2, 2014 at 12:04 PM, Akhil Das <akhil@sigmoidanalytics.com> wrote:
What is the application that you are submitting? Looks like you might have invoked fs inside the app and then closed it within it.

Thanks
Best Regards

On Tue, Dec 2, 2014 at 11:59 AM, rapelly kartheek <kartheek.mbms@gmail.com> wrote:
Hi,

I face the following exception when submit a spark application. The log file shows:

14/12/02 11:52:58 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1668)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1629)
at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1614)
at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:120)
at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:158)
at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:158)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.util.FileLogger.flush(FileLogger.scala:158)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:87)
at org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:112)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$4.apply(SparkListenerBus.scala:52)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$4.apply(SparkListenerBus.scala:52)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$foreachListener$1.apply(SparkListenerBus.scala:81)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$foreachListener$1.apply(SparkListenerBus.scala:79)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.SparkListenerBus$class.foreachListener(SparkListenerBus.scala:79)
at org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:52)
at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:32)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:56)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:56)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:56)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1160)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:46)

Someone please help me resolve this!!

Thanks