spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mufy <mufeed.us...@gmail.com>
Subject Re: Setting Spark TMP Directory in Cluster Mode
Date Wed, 30 Sep 2015 07:38:14 GMT
Any takers? :-)


---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>




On Mon, Sep 28, 2015 at 10:19 AM, mufy <mufeed.usman@gmail.com> wrote:

> Hello Akhil,
>
> I do not see how that would work for a YARN cluster mode execution
> since the local directories used by the Spark executors and the Spark
> driver are the local directories that are configured for YARN
> (yarn.nodemanager.local-dirs). If you specify a different path with
> SPARK_LOCAL_DIRS, that path will be ignored.
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>

Mime
View raw message