spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bui, Tri" <>
Subject RE: Read a HDFS file from Spark using HDFS API
Date Fri, 14 Nov 2014 15:48:02 GMT
It should be

val file = sc.textFile("hdfs:///localhost:9000/sigmoid/input.txt")

3 “///”


From: rapelly kartheek []
Sent: Friday, November 14, 2014 9:42 AM
To: Akhil Das;
Subject: Re: Read a HDFS file from Spark using HDFS API

No. I am not accessing hdfs from either shell or a spark application. I want to access from
spark "Scheduler code".

I face an error when I use sc.textFile() as SparkContext wouldn't have been created yet. So,
error says: "sc not found".

On Fri, Nov 14, 2014 at 9:07 PM, Akhil Das <<>>
like this?

val file = sc.textFile("hdfs://localhost:9000/sigmoid/input.txt")

Best Regards

On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek <<>>
I am trying to read a HDFS file from Spark "scheduler code". I could find how to write hdfs
read/writes in java.

But I  need to access hdfs from spark using scala. Can someone please help me in this regard.

View raw message