spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mufy <mufeed.us...@gmail.com>
Subject Re: Setting Spark TMP Directory in Cluster Mode
Date Mon, 28 Sep 2015 04:49:32 GMT
Hello Akhil,

I do not see how that would work for a YARN cluster mode execution
since the local directories used by the Spark executors and the Spark
driver are the local directories that are configured for YARN
(yarn.nodemanager.local-dirs). If you specify a different path with
SPARK_LOCAL_DIRS, that path will be ignored.


---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>

Mime
View raw message