spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dino Fancellu <d...@felstar.com>
Subject Local Spark talking to remote HDFS?
Date Mon, 24 Aug 2015 18:46:05 GMT
I have a file in HDFS inside my HortonWorks HDP 2.3_1 VirtualBox VM.

If I go into the guest spark-shell and refer to the file thus, it works fine

  val words=sc.textFile("hdfs:///tmp/people.txt")
  words.count

However if I try to access it from a local Spark app on my Windows host, it
doesn't work

  val conf = new SparkConf().setMaster("local").setAppName("My App")
  val sc = new SparkContext(conf)
  
  val words=sc.textFile("hdfs://localhost:8020/tmp/people.txt")
  words.count

Emits



The port 8020 is open, and if I choose the wrong file name, it will tell me



My pom has

	<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-core_2.11</artifactId>
			<version>1.4.1</version>
			<scope>provided</scope>
		</dependency>

Am I doing something wrong?

Thanks.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Local-Spark-talking-to-remote-HDFS-tp24425.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message