spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <>
Subject Re: Setup Remote HDFS for Spark
Date Tue, 25 Nov 2014 01:21:00 GMT

On Sat, Nov 22, 2014 at 12:13 AM, EH <> wrote:

> Unfortunately whether it is possible to have both Spark and HDFS running on
> the same machine is not under our control.  :(  Right now we have Spark and
> HDFS running in different machines.  In this case, is it still possible to
> hook up a remote HDFS with Spark so that we can use Spark Streaming
> checkpoints?  Thank you for your help.

In my case, after I copied my cluster's core-site.xml, yarn-site.xml, and
hdfs-site.xml to my CLASSPATH, I could access YARN and HDFS remotely
without any problems (read: modulo network/firewall issues), which I was
pretty surprised by myself.


View raw message