spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexei Bakanov <russ...@gmail.com>
Subject Temp files are not removed when done (Mesos)
Date Wed, 07 Oct 2015 14:14:13 GMT
Hi

I'm running Spark 1.5.1 on Mesos in coarse-grained mode. No dynamic
allocation or shuffle service. I see that there are two types of temporary
files under /tmp folder associated with every executor: /tmp/spark-<UUID>
and /tmp/blockmgr-<UUID>. When job is finished /tmp/spark-<UUID> is gone,
but blockmgr directory is left with all gigabytes in it. In Spark 1.4.1
blockmgr-<UUID> folder was under /tmp/spark-<UUID>, so when /tmp/spark
folder was removed blockmgr was gone too.
Is it a bug in 1.5.1?

By the way, in fine-grain mode /tmp/spark-<UUID> folder does not get
removed in neither 1.4.1 nor 1.5.1 for some reason.

Thanks,
Alexei

Mime
View raw message