spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Baretta <>
Subject /tmp directory fills up
Date Sat, 10 Jan 2015 07:32:00 GMT

I'm building spark using the current master branch and deploying in to
Google Compute Engine on top of Hadoop 2.4/YARN via bdutil, Google's Hadoop
cluster provisioning tool. bdutils configures Spark with


but this option is ignored in combination with YARN. Bdutils also
configures YARN with:

      Directories on the local machine in which to application temp files.

This is the right directory for spark to store temporary data in. Still,
Spark is creating such directories as this:


and filling them up with gigabytes worth of output files, filling up the
very small root filesystem.

How can I diagnose why my Spark installation is not picking up the
yarn.nodemanager.local-dirs from yarn?


View raw message