spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuporno Choudhury <>
Subject Clearing usercache on EMR [pyspark]
Date Wed, 01 Aug 2018 07:19:47 GMT
Hi everyone,
I am running spark jobs on EMR (using pyspark). I noticed that after
running jobs, the size of the usercache (basically the filecache folder)
keeps on increasing (with directory names as 1,2,3,4,5,...).
    Directory location: */mnt/yarn/usercache/hadoop/**filecache/*
Is there a way to avoid creating these directories or automatically
clearing the usercache/filecache after a job/periodically?
Shuporno Choudhury

View raw message