spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghavendra Pandey <>
Subject Re: Cleaning up spark.local.dir automatically
Date Fri, 09 Jan 2015 15:28:30 GMT
You may like to look at spark.cleaner.ttl configuration which is infinite
by default. Spark has that configuration to delete temp files time to time.

On Fri Jan 09 2015 at 8:34:10 PM <> wrote:

>  Hi,
> Is there a way of automatically cleaning up the spark.local.dir after a
> job has been run? I have noticed a large number of temporary files have
> been stored here and are not cleaned up. The only solution I can think of
> is to run some sort of cron job to delete files older than a few days. I am
> currently using a mixture of standalone and YARN spark builds.
> Thanks,
> Michael
> This e-mail (including any attachments) is private and confidential, may
> contain proprietary or privileged information and is intended for the named
> recipient(s) only. Unintended recipients are strictly prohibited from
> taking action on the basis of information in this e-mail and must contact
> the sender immediately, delete this e-mail (and all attachments) and
> destroy any hard copies. Nomura will not accept responsibility or liability
> for the accuracy or completeness of, or the presence of any virus or
> disabling code in, this e-mail. If verification is sought please request a
> hard copy. Any reference to the terms of executed transactions should be
> treated as preliminary only and subject to formal written confirmation by
> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail
> communications through its networks (subject to and in accordance with
> applicable laws). No confidentiality or privilege is waived or lost by
> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is
> a reference to any entity in the Nomura Holdings, Inc. group. Please read
> our Electronic Communications Legal Notice which forms part of this e-mail:

View raw message