spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng, Hao" <hao.ch...@intel.com>
Subject RE: Question on saveAsTextFile with overwrite option
Date Thu, 25 Dec 2014 07:28:45 GMT
I am wondering if we can provide more friendly API, other than configuration for this purpose.
What do you think Patrick?

Cheng Hao

-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com] 
Sent: Thursday, December 25, 2014 3:22 PM
To: Shao, Saisai
Cc: user@spark.apache.org; dev@spark.apache.org
Subject: Re: Question on saveAsTextFile with overwrite option

Is it sufficient to set "spark.hadoop.validateOutputSpecs" to false?

http://spark.apache.org/docs/latest/configuration.html

- Patrick

On Wed, Dec 24, 2014 at 10:52 PM, Shao, Saisai <saisai.shao@intel.com> wrote:
> Hi,
>
>
>
> We have such requirements to save RDD output to HDFS with 
> saveAsTextFile like API, but need to overwrite the data if existed. 
> I'm not sure if current Spark support such kind of operations, or I need to check this
manually?
>
>
>
> There's a thread in mailing list discussed about this 
> (http://apache-spark-user-list.1001560.n3.nabble.com/How-can-I-make-Sp
> ark-1-0-saveAsTextFile-to-overwrite-existing-file-td6696.html),
> I'm not sure this feature is enabled or not, or with some configurations?
>
>
>
> Appreciate your suggestions.
>
>
>
> Thanks a lot
>
> Jerry

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail:
user-help@spark.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message