spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rapelly kartheek <kartheek.m...@gmail.com>
Subject Re: java.io.IOException: Filesystem closed
Date Tue, 02 Dec 2014 08:55:58 GMT
Does the sparkContext shuts down itself by default even if I dont mention
specifically in my code?? Because, I ran the application without
sc.context(), still I get file system closed error along with correct
output.

On Tue, Dec 2, 2014 at 2:20 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> It could be because those threads are finishing quickly.
>
> Thanks
> Best Regards
>
> On Tue, Dec 2, 2014 at 2:19 PM, rapelly kartheek <kartheek.mbms@gmail.com>
> wrote:
>
>> But, somehow, if I run this application for the second time, I find that
>> the application gets executed and the results are out regardless of the
>> same errors in logs.
>>
>> On Tue, Dec 2, 2014 at 2:08 PM, Akhil Das <akhil@sigmoidanalytics.com>
>> wrote:
>>
>>> Your code seems to have a lot of threads and i think you might be
>>> invoking sc.stop before those threads get finished.
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Tue, Dec 2, 2014 at 12:04 PM, Akhil Das <akhil@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> What is the application that you are submitting? Looks like you might
>>>> have invoked fs inside the app and then closed it within it.
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Tue, Dec 2, 2014 at 11:59 AM, rapelly kartheek <
>>>> kartheek.mbms@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I face the following exception when submit a spark application. The
>>>>> log file shows:
>>>>>
>>>>> 14/12/02 11:52:58 ERROR LiveListenerBus: Listener EventLoggingListener
>>>>> threw an exception
>>>>> java.io.IOException: Filesystem closed
>>>>> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1668)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1629)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1614)
>>>>> at
>>>>> org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:120)
>>>>> at
>>>>> org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:158)
>>>>> at
>>>>> org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:158)
>>>>> at scala.Option.foreach(Option.scala:236)
>>>>> at org.apache.spark.util.FileLogger.flush(FileLogger.scala:158)
>>>>> at
>>>>> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:87)
>>>>> at
>>>>> org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:112)
>>>>> at
>>>>> org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$4.apply(SparkListenerBus.scala:52)
>>>>> at
>>>>> org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$4.apply(SparkListenerBus.scala:52)
>>>>> at
>>>>> org.apache.spark.scheduler.SparkListenerBus$$anonfun$foreachListener$1.apply(SparkListenerBus.scala:81)
>>>>> at
>>>>> org.apache.spark.scheduler.SparkListenerBus$$anonfun$foreachListener$1.apply(SparkListenerBus.scala:79)
>>>>> at
>>>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>>>> at
>>>>> org.apache.spark.scheduler.SparkListenerBus$class.foreachListener(SparkListenerBus.scala:79)
>>>>> at
>>>>> org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:52)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:32)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:56)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:56)
>>>>> at scala.Option.foreach(Option.scala:236)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:56)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
>>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1160)
>>>>> at
>>>>> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:46)
>>>>>
>>>>> Someone please help me resolve this!!
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message