Please check the driver and executor log, there should be logs about where the data is written.--On Wed, Jun 22, 2016 at 2:03 AM, Pierre Villard <email@example.com> wrote:
I have a Spark job writing files to HDFS using .saveAsHadoopFile method.
If I run my job in local/client mode, it works as expected and I get all my files written in HDFS. However if I change to yarn/cluster mode, I don't see any error logs (the job is successful) and there is no files written to HDFS.
Is there any reason for this behavior? Any thoughts on how to track down what is happening here?