spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rapelly kartheek <kartheek.m...@gmail.com>
Subject Re: Read a HDFS file from Spark using HDFS API
Date Fri, 14 Nov 2014 15:59:13 GMT
I'll just try out with object Akhil provided.
There was no problem working in shell with sc.textFile.

Thank you Akhil and Tri.

On Fri, Nov 14, 2014 at 9:21 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> [image: Inline image 1]
>
>
> Thanks
> Best Regards
>
> On Fri, Nov 14, 2014 at 9:18 PM, Bui, Tri <
> Tri.Bui@verizonwireless.com.invalid> wrote:
>
>> It should be
>>
>>
>>
>> val file = sc.textFile("hdfs:///localhost:9000/sigmoid/input.txt")
>>
>>
>>
>> 3 “///”
>>
>>
>>
>> Thanks
>>
>> Tri
>>
>>
>>
>> *From:* rapelly kartheek [mailto:kartheek.mbms@gmail.com]
>> *Sent:* Friday, November 14, 2014 9:42 AM
>> *To:* Akhil Das; user@spark.apache.org
>> *Subject:* Re: Read a HDFS file from Spark using HDFS API
>>
>>
>>
>> No. I am not accessing hdfs from either shell or a spark application. I
>> want to access from spark "Scheduler code".
>>
>>
>>
>> I face an error when I use sc.textFile() as SparkContext wouldn't have
>> been created yet. So, error says: "sc not found".
>>
>>
>>
>> On Fri, Nov 14, 2014 at 9:07 PM, Akhil Das <akhil@sigmoidanalytics.com>
>> wrote:
>>
>> like this?
>>
>>
>>
>> val file = sc.textFile("hdfs://localhost:9000/sigmoid/input.txt")
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>
>> Hi,
>>
>> I am trying to read a HDFS file from Spark "scheduler code". I could find
>> how to write hdfs read/writes in java.
>>
>>
>>
>> But I  need to access hdfs from spark using scala. Can someone please
>> help me in this regard.
>>
>>
>>
>>
>>
>
>

Mime
View raw message