spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen, Kevin" <Kevin.C...@neustar.biz>
Subject Missing output partition file in S3
Date Thu, 15 Sep 2016 18:37:01 GMT
Hi,

Has any one encountered an issue of missing output partition file in S3 ? My spark job writes
output to a S3 location. Occasionally, I noticed one partition file is missing. As a result,
one chunk of data was lost. If I rerun the same job, the problem usually goes away. This has
been happening pretty random. I observed once or twice a week on a daily run job. I am using
Spark 1.2.1.

Very much appreciated on any input, suggestion of fix/workaround.




Mime
View raw message