spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Mistroni <>
Subject Problem in persisting file in S3 using Spark: xxx file does not exist Exception
Date Tue, 24 Apr 2018 21:28:46 GMT
HI all
 i am using the following code for persisting data into S3 (aws keys are
already stored in the environment variables)


However, i keep on receiving an exception that the file does not exist

here's what comes from logs

18/04/24 22:15:32 INFO Persiste: Persisting data to text file:
Exception in thread "main" /form4-results-2404.results
doesn't exist

It seems that Spark expects the file to be there before writing? which
seems bizzarre?

I Have even tried to remove the coalesce ,but still got the same exception
Could anyone help pls?
kind regarsd

View raw message