spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Cleaning up spark.local.dir automatically
Date Fri, 09 Jan 2015 22:42:08 GMT
That's a worker setting which cleans up the files left behind by executors,
so spark.cleaner.ttl isn't at the RDD level.  After
https://issues.apache.org/jira/browse/SPARK-1860 the cleaner won't clean up
directories left by running executors.


On Fri, Jan 9, 2015 at 7:38 AM, <michael.england@nomura.com> wrote:

>  Thanks, I imagine this will kill any cached RDDs if their files are
> beyond the ttl?
>
>
>
> Thanks
>
>
>
>
>
> *From:* Raghavendra Pandey [mailto:raghavendra.pandey@gmail.com]
> *Sent:* 09 January 2015 15:29
> *To:* England, Michael (IT/UK); user@spark.apache.org
> *Subject:* Re: Cleaning up spark.local.dir automatically
>
>
>
> You may like to look at spark.cleaner.ttl configuration which is infinite
> by default. Spark has that configuration to delete temp files time to time.
>
> On Fri Jan 09 2015 at 8:34:10 PM <michael.england@nomura.com> wrote:
>
> Hi,
>
>
>
> Is there a way of automatically cleaning up the spark.local.dir after a
> job has been run? I have noticed a large number of temporary files have
> been stored here and are not cleaned up. The only solution I can think of
> is to run some sort of cron job to delete files older than a few days. I am
> currently using a mixture of standalone and YARN spark builds.
>
>
>
> Thanks,
>
> Michael
>
>
>
>
>
> This e-mail (including any attachments) is private and confidential, may
> contain proprietary or privileged information and is intended for the named
> recipient(s) only. Unintended recipients are strictly prohibited from
> taking action on the basis of information in this e-mail and must contact
> the sender immediately, delete this e-mail (and all attachments) and
> destroy any hard copies. Nomura will not accept responsibility or liability
> for the accuracy or completeness of, or the presence of any virus or
> disabling code in, this e-mail. If verification is sought please request a
> hard copy. Any reference to the terms of executed transactions should be
> treated as preliminary only and subject to formal written confirmation by
> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail
> communications through its networks (subject to and in accordance with
> applicable laws). No confidentiality or privilege is waived or lost by
> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is
> a reference to any entity in the Nomura Holdings, Inc. group. Please read
> our Electronic Communications Legal Notice which forms part of this e-mail:
> http://www.Nomura.com/email_disclaimer.htm
>
> This e-mail (including any attachments) is private and confidential, may
> contain proprietary or privileged information and is intended for the named
> recipient(s) only. Unintended recipients are strictly prohibited from
> taking action on the basis of information in this e-mail and must contact
> the sender immediately, delete this e-mail (and all attachments) and
> destroy any hard copies. Nomura will not accept responsibility or liability
> for the accuracy or completeness of, or the presence of any virus or
> disabling code in, this e-mail. If verification is sought please request a
> hard copy. Any reference to the terms of executed transactions should be
> treated as preliminary only and subject to formal written confirmation by
> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail
> communications through its networks (subject to and in accordance with
> applicable laws). No confidentiality or privilege is waived or lost by
> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is
> a reference to any entity in the Nomura Holdings, Inc. group. Please read
> our Electronic Communications Legal Notice which forms part of this e-mail:
> http://www.Nomura.com/email_disclaimer.htm
>

Mime
View raw message