spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: how to clean shuffle write each iteration
Date Tue, 03 Mar 2015 07:18:53 GMT
Can't find anything related to this from the Configurations page
http://spark.apache.org/docs/1.2.0/configuration.html, You could probably
open a JIRA issue regarding this.

Thanks
Best Regards

On Tue, Mar 3, 2015 at 12:03 PM, lisendong <lisendong@163.com> wrote:

> I 'm using spark als.
>
> I set the iteration number to 30.
>
> And in each iteration, tasks will produce nearly 1TB shuffle write.
>
> To my surprise, this shuffle data will not be cleaned until the total job
> finished, which means, I need 30TB disk to store the shuffle data.
>
>
> I think after each iteration, we can delete the shuffle data before current
> iteration, right?
>
> how to do this?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-clean-shuffle-write-each-iteration-tp21886.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message