spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1407) EventLogging to HDFS doesn't work properly on yarn
Date Tue, 08 Apr 2014 19:01:15 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963306#comment-13963306
] 

Thomas Graves commented on SPARK-1407:
--------------------------------------

I was just running SparkHdfsLR.  I also tried JavaWordCount, same issue.  SparkPi works since
it doesn't access hdfs. 

> EventLogging to HDFS doesn't work properly on yarn
> --------------------------------------------------
>
>                 Key: SPARK-1407
>                 URL: https://issues.apache.org/jira/browse/SPARK-1407
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Thomas Graves
>            Priority: Blocker
>
> When running on spark on yarn and accessing an HDFS file (like in the SparkHdfsLR example)
while using the event logging configured to write logs to HDFS, it throws an exception at
the end of the application. 
> SPARK_JAVA_OPTS=-Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs:///history/spark/
> 14/04/03 13:41:31 INFO yarn.ApplicationMaster$$anon$1: Invoking sc stop from shutdown
hook
> Exception in thread "Thread-41" java.io.IOException: Filesystem closed
>         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:398)
>         at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1465)
>         at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1450)
>         at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:116)
>         at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
>         at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
>         at scala.Option.foreach(Option.scala:236)
>         at org.apache.spark.util.FileLogger.flush(FileLogger.scala:137)
>         at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:69)
>         at org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:101)
>         at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
>         at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
>         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>         at org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:67)
>         at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:31)
>         at org.apache.spark.scheduler.LiveListenerBus.post(LiveListenerBus.scala:78)
>         at org.apache.spark.SparkContext.postApplicationEnd(SparkContext.scala:1081)
>         at org.apache.spark.SparkContext.stop(SparkContext.scala:828)
>         at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:460)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message