spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankits <ankitso...@gmail.com>
Subject spark.cleaner questions
Date Tue, 13 Jan 2015 21:41:33 GMT
I am using spark 1.1 with the ooyala job server (which basically creates long
running spark jobs as contexts to execute jobs in). These contexts have
cached RDDs in memory (via RDD.persist()).

I want to enable the spark.cleaner to cleanup the /spark/work directories
that are created for each app, but not touch cached RDDs like so:

spark.worker.cleanup.enabled = true
spark.worker.cleanup.interval = 1800
spark.worker.cleanup.appDataTtl = 604800 #7 days

2 questions here -
Will these settings affect cleanup of cached RDDs? (because I want those to
be persisted forever)

Is there a way to force the cleaner to run, and how can I see when the
cleaner is run? After settings these options, I still see the data for apps
older than 7 days on the worker nodes. Why is that happening?





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-cleaner-questions-tp21128.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message