spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mufy <>
Subject Re: Setting Spark TMP Directory in Cluster Mode
Date Mon, 28 Sep 2015 04:49:32 GMT
Hello Akhil,

I do not see how that would work for a YARN cluster mode execution
since the local directories used by the Spark executors and the Spark
driver are the local directories that are configured for YARN
(yarn.nodemanager.local-dirs). If you specify a different path with
SPARK_LOCAL_DIRS, that path will be ignored.

Mufeed Usman
My LinkedIn <> | My
Social Cause <> | My Blogs : LiveJournal

View raw message