spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Das <debasish.da...@gmail.com>
Subject SPARK_LOCAL_DIRS option
Date Thu, 14 Aug 2014 01:47:53 GMT
Hi,

I have set up the SPARK_LOCAL_DIRS option in spark-env.sh so that Spark can
use more shuffle space...

Does Spark cleans all the shuffle files once the runs are done ? Seems to
me that the shuffle files are not cleaned...

Do I need to set this variable ? spark.cleaner.ttl

Right now we are planning to use logRotate to clean up the shuffle
files..is that a good practice ?

Thanks.
Deb

Mime
View raw message