spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francis.Hu <francis...@reachjunction.com>
Subject 答复: 答复: java.io.FileNotFoundException: /test/spark-0.9.1/work/app-20140505053550-0000/2/stdout (No such file or directory)
Date Tue, 06 May 2014 02:31:15 GMT
i looked into the log again, all exceptions are about FileNotFoundException . In the Webui,
no anymore info I can check except for the basic description of job.  

Attached the log file, could you help to take a look ? Thanks.

 

Francis.Hu

 

发件人: Tathagata Das [mailto:tathagata.das1565@gmail.com] 
发送时间: Tuesday, May 06, 2014 10:16
收件人: user@spark.apache.org
主题: Re: 答复: java.io.FileNotFoundException: /test/spark-0.9.1/work/app-20140505053550-0000/2/stdout
(No such file or directory)

 

Can you check the Spark worker logs on that machine. Either from the web ui, or directly.
Should be /test/spark-XXX/logs/  See if that has any error.

If there is not permission issue, I am not why stdout and stderr is not being generated. 

 

TD

 

On Mon, May 5, 2014 at 7:13 PM, Francis.Hu <francis.hu@reachjunction.com> wrote:

The file does not exist in fact and no permission issue. 

 

francis@ubuntu-4:/test/spark-0.9.1$ ll work/app-20140505053550-0000/

total 24

drwxrwxr-x  6 francis francis 4096 May  5 05:35 ./

drwxrwxr-x 11 francis francis 4096 May  5 06:18 ../

drwxrwxr-x  2 francis francis 4096 May  5 05:35 2/

drwxrwxr-x  2 francis francis 4096 May  5 05:35 4/

drwxrwxr-x  2 francis francis 4096 May  5 05:35 7/

drwxrwxr-x  2 francis francis 4096 May  5 05:35 9/

 

Francis

 

发件人: Tathagata Das [mailto:tathagata.das1565@gmail.com] 
发送时间: Tuesday, May 06, 2014 3:45
收件人: user@spark.apache.org
主题: Re: java.io.FileNotFoundException: /test/spark-0.9.1/work/app-20140505053550-0000/2/stdout
(No such file or directory)

 

Do those file actually exist? Those stdout/stderr should have the output of the spark's executors
running in the workers, and its weird that they dont exist. Could be permission issue - maybe
the directories/files are not being generated because it cannot?

 

TD

 

On Mon, May 5, 2014 at 3:06 AM, Francis.Hu <francis.hu@reachjunction.com> wrote:

Hi,All

 

 

We run a spark cluster with three workers. 

created a spark streaming application,

then run the spark project using below command:

 

shell> sbt run spark://192.168.219.129:7077 tcp://192.168.20.118:5556 foo

 

we looked at the webui of workers, jobs failed without any error or info, but FileNotFoundException
occurred in workers' log file as below:

Is this an existent issue of spark? 

 

 

-----------------------------in workers' logs/spark-francis-org.apache.spark.deploy.worker.Worker-1-ubuntu-4.out----------------------------

 

14/05/05 02:39:39 WARN AbstractHttpConnection: /logPage/?appId=app-20140505053550-0000&executorId=2&logType=stdout

java.io.FileNotFoundException: /test/spark-0.9.1/work/app-20140505053550-0000/2/stdout (No
such file or directory)

        at java.io.FileInputStream.open(Native Method)

        at java.io.FileInputStream.<init>(FileInputStream.java:138)

        at org.apache.spark.util.Utils$.offsetBytes(Utils.scala:687)

        at org.apache.spark.deploy.worker.ui.WorkerWebUI.logPage(WorkerWebUI.scala:119)

        at org.apache.spark.deploy.worker.ui.WorkerWebUI$$anonfun$6.apply(WorkerWebUI.scala:52)

        at org.apache.spark.deploy.worker.ui.WorkerWebUI$$anonfun$6.apply(WorkerWebUI.scala:52)

        at org.apache.spark.ui.JettyUtils$$anon$1.handle(JettyUtils.scala:61)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1040)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:976)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.Server.handle(Server.java:363)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)

        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

        at java.lang.Thread.run(Thread.java:722)

14/05/05 02:39:41 WARN AbstractHttpConnection: /logPage/?appId=app-20140505053550-0000&executorId=9&logType=stderr

java.io.FileNotFoundException: /test/spark-0.9.1/work/app-20140505053550-0000/9/stderr (No
such file or directory)

        at java.io.FileInputStream.open(Native Method)

        at java.io.FileInputStream.<init>(FileInputStream.java:138)

        at org.apache.spark.util.Utils$.offsetBytes(Utils.scala:687)

        at org.apache.spark.deploy.worker.ui.WorkerWebUI.logPage(WorkerWebUI.scala:119)

        at org.apache.spark.deploy.worker.ui.WorkerWebUI$$anonfun$6.apply(WorkerWebUI.scala:52)

        at org.apache.spark.deploy.worker.ui.WorkerWebUI$$anonfun$6.apply(WorkerWebUI.scala:52)

        at org.apache.spark.ui.JettyUtils$$anon$1.handle(JettyUtils.scala:61)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1040)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:976)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.Server.handle(Server.java:363)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)

        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

        at java.lang.Thread.run(Thread.java:722)

 

 

 

Thanks,

Francis.Hu

 

 


Mime
View raw message