spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From censj <ce...@lotuseed.com>
Subject Re: FileNotFoundException, while file is actually available
Date Mon, 06 Feb 2017 07:58:45 GMT
If you deploy yarn model,you can used yarn logs -applicationId youApplicationId get yarn
logs. You can get logs in details。
Then Looking error info,
===============================
Name: cen sujun
Mobile: 13067874572
Mail: censj@lotuseed.com

> 在 2017年2月6日,05:33,Evgenii Morozov <evgeny.a.morozov@gmail.com> 写道:
> 
> Hi, 
> 
> I see a lot of exceptions like the following during our machine learning pipeline calculation.
Spark version 2.0.2.
> Sometimes it’s just few executors that fails with this message, but the job is successful.

> 
> I’d appreciate any hint you might have.
> Thank you.
> 
> 2017-02-05 07:56:47.022 [task-result-getter-1] WARN  o.a.spark.scheduler.TaskSetManager
- Lost task 0.0 in stage 151558.0 (TID 993070, 10.61.12.43):
> java.io.FileNotFoundException: File file:/path/to/file does not exist
>        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
>        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
>        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
>        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
>        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
>        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
>        at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:109)
>        at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
>        at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:245)
>        at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>        at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>        at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>        at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>        at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>        at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>        at org.apache.spark.scheduler.Task.run(Task.scala:86)
>        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>        at java.lang.Thread.run(Thread.java:745)
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> 


Mime
View raw message