Without (C), what is the best practice to implement the following scenario?
1. rdd = sc.textFile(FileA)
2. rdd = rdd.map(...) // actually modifying the rdd
Since the rdd transformation is 'lazy', rdd will not materialize until
saveAsTextFile(), so FileA must still exist, but it must be deleted before
What I can think is:
4. delete FileA
5. rename TempFile to FileA
This is not very convenient...
From: Patrick Wendell [mailto:email@example.com]
Sent: Tuesday, June 03, 2014 11:40 AM
Subject: Re: How can I make Spark 1.0 saveAsTextFile to overwrite existing
(A) Semantics in Spark 0.9 and earlier: Spark will ignore Hadoo's output
format check and overwrite files in the destination directory.
But it won't clobber the directory entirely. I.e. if the directory already
had "part1" "part2" "part3" "part4" and you write a new job outputing only
two files ("part1", "part2") then it would leave the other two files intact,
(B) Semantics in Spark 1.0 and earlier: Runs Hadoop OutputFormat check which
means the directory must not exist already or an excpetion is thrown.
(C) Semantics proposed by Nicholas Chammas in this thread (AFAIK):
Spark will delete/clobber an existing destination directory if it exists,
then fully over-write it with new data.
I'm fine to add a flag that allows (B) for backwards-compatibility reasons,
but my point was I'd prefer not to have (C) even though I see some cases
where it would be useful.
On Mon, Jun 2, 2014 at 4:25 PM, Sean Owen <firstname.lastname@example.org> wrote:
> Is there a third way? Unless I miss something. Hadoop's OutputFormat
> wants the target dir to not exist no matter what, so it's just a
> question of whether Spark deletes it for you or errors.
> On Tue, Jun 3, 2014 at 12:22 AM, Patrick Wendell <email@example.com>
>> We can just add back a flag to make it backwards compatible - it was
>> just missed during the original PR.
>> Adding a *third* set of "clobber" semantics, I'm slightly -1 on that
>> for the following reasons:
>> 1. It's scary to have Spark recursively deleting user files, could
>> easily lead to users deleting data by mistake if they don't
>> understand the exact semantics.
>> 2. It would introduce a third set of semantics here for saveAsXX...
>> 3. It's trivial for users to implement this with two lines of code
>> (if output dir exists, delete it) before calling saveAsHadoopFile.
>> - Patrick