spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Does saveAsHadoopFile depend on master?
Date Wed, 22 Jun 2016 01:57:54 GMT
Please check the driver and executor log, there should be logs about where
the data is written.



On Wed, Jun 22, 2016 at 2:03 AM, Pierre Villard <pierre.villard.fr@gmail.com
> wrote:

> Hi,
>
> I have a Spark job writing files to HDFS using .saveAsHadoopFile method.
>
> If I run my job in local/client mode, it works as expected and I get all
> my files written in HDFS. However if I change to yarn/cluster mode, I don't
> see any error logs (the job is successful) and there is no files written to
> HDFS.
>
> Is there any reason for this behavior? Any thoughts on how to track down
> what is happening here?
>
> Thanks!
>
> Pierre.
>



-- 
Best Regards

Jeff Zhang

Mime
View raw message