spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Trying to find where Spark persists RDDs when run with YARN
Date Sun, 18 Jan 2015 18:21:42 GMT
These will be under the working directory of the YARN container
running the executor. I don't have it handy but think it will also be
a "spark-local" or similar directory.

On Sun, Jan 18, 2015 at 2:50 PM, Hemanth Yamijala <yhemanth@gmail.com> wrote:
> Hi,
>
> I am trying to find where Spark persists RDDs when we call the persist() api
> and executed under YARN. This is purely for understanding...
>
> In my driver program, I wait indefinitely, so as to avoid any clean up
> problems.
>
> In the actual job, I roughly do the following:
>
> JavaRDD<String> lines = context.textFile(args[0]);
> lines.persist(StorageLevel.DISK_ONLY());
> lines.collect();
>
> When run with local executor, I can see that the files (like rdd_1_0) are
> persisted under directories like
> /var/folders/mt/51srrjc15wl3n829qkgnh2dm0000gp/T/spark-local-20150118201458-6147/15.
>
> Where similarly can I find these under Yarn ?
>
> Thanks
> hemanth

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message