spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Bhatnagar <aniket.bhatna...@gmail.com>
Subject Spark doesn't retry task while writing to HDFS
Date Fri, 24 Oct 2014 13:45:52 GMT
Hi all

I have written a job that reads data from HBASE and writes to HDFS (fairly
simple). While running the job, I noticed that a few of the tasks failed
with the following error. Quick googling on the error suggests that its an
unexplained error and is perhaps intermittent. What I am curious to know is
why didn't Spark retry writing file to HDFS? It just shows it as failed job
in Spark UI.

Error:
java.io.IOException: All datanodes x.x.x.x:yyyy are bad. Aborting...

org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1128)

org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)

org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)


Thanks,
Aniket

Mime
View raw message