spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankits <>
Subject spark.cleaner questions
Date Tue, 13 Jan 2015 21:41:33 GMT
I am using spark 1.1 with the ooyala job server (which basically creates long
running spark jobs as contexts to execute jobs in). These contexts have
cached RDDs in memory (via RDD.persist()).

I want to enable the spark.cleaner to cleanup the /spark/work directories
that are created for each app, but not touch cached RDDs like so:

spark.worker.cleanup.enabled = true
spark.worker.cleanup.interval = 1800
spark.worker.cleanup.appDataTtl = 604800 #7 days

2 questions here -
Will these settings affect cleanup of cached RDDs? (because I want those to
be persisted forever)

Is there a way to force the cleaner to run, and how can I see when the
cleaner is run? After settings these options, I still see the data for apps
older than 7 days on the worker nodes. Why is that happening?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message