spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JF Chen <darou...@gmail.com>
Subject Re: how to change temp directory when spark write data ?
Date Wed, 05 Dec 2018 13:50:26 GMT
Directory to use for "scratch" space in Spark, including map output files
and RDDs that get stored on disk.
It seems a directory on my local disk. But currently the temp directory is
on hdfs, under the path of  df.write.parquet(path)

Regard,
Junfeng Chen


On Wed, Dec 5, 2018 at 6:28 PM Sandip Mehta <sandip.mehta.sub@gmail.com>
wrote:

> tryspark.local.dir property.
>
>
> On Wed, Dec 5, 2018 at 1:42 PM JF Chen <darouwan@gmail.com> wrote:
>
>> I have two spark apps writing data to one directory. I notice they share
>> one temp directory, and the spark fist finish writing will clear the temp
>> directory and the slower one may throw "No lease on *** File does not
>> exist" error
>> So how to specify the temp directory?
>> Thanks!
>>
>> Regard,
>> Junfeng Chen
>>
>

Mime
View raw message