spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuporno Choudhury <shuporno.choudh...@gmail.com>
Subject Re: Clearing usercache on EMR [pyspark]
Date Fri, 03 Aug 2018 06:43:51 GMT
 Can anyone please help me with this issue?

On Fri, 3 Aug 2018 at 11:27, Shuporno Choudhury <
shuporno.choudhury@gmail.com> wrote:

> Can anyone please help me with this issue?
>
> On Wed, 1 Aug 2018 at 12:50, Shuporno Choudhury [via Apache Spark User
> List] <ml+s1001560n33096h62@n3.nabble.com> wrote:
>
>> Hi everyone,
>> I am running spark jobs on EMR (using pyspark). I noticed that after
>> running jobs, the size of the usercache (basically the filecache folder)
>> keeps on increasing (with directory names as 1,2,3,4,5,...).
>>     Directory location: */mnt/yarn/usercache/hadoop/**filecache/*
>> Is there a way to avoid creating these directories or automatically
>> clearing the usercache/filecache after a job/periodically?
>> --
>> --Thanks,
>> Shuporno Choudhury
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Clearing-usercache-on-EMR-pyspark-tp33096.html
>> To start a new topic under Apache Spark User List, email
>> ml+s1001560n1h17@n3.nabble.com
>> To unsubscribe from Apache Spark User List, click here
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=c2h1cG9ybm8uY2hvdWRodXJ5QGdtYWlsLmNvbXwxfC0xODI0MTU0MzQ0>
>> .
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> --
> --Thanks,
> Shuporno Choudhury
>


-- 
--Thanks,
Shuporno Choudhury

Mime
View raw message