i am using the following code for persisting data into S3 (aws keys are already stored in the environment variables)
However, i keep on receiving an exception that the file does not exist
here's what comes from logs
18/04/24 22:15:32 INFO Persiste: Persisting data to text file: s3://ec2-bucket-mm-spark/form4-results-2404.results
Exception in thread "main" java.io.IOException: /form4-results-2404.results doesn't exist
It seems that Spark expects the file to be there before writing? which seems bizzarre?
I Have even tried to remove the coalesce ,but still got the same exception
Could anyone help pls?