spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajay Chander <itsche...@gmail.com>
Subject Spark_API_Copy_From_Edgenode
Date Fri, 27 May 2016 16:27:55 GMT
Hi Everyone,

                           I have some data located on the EdgeNode. Right
now, the process I follow to copy the data from Edgenode to HDFS is through
a shellscript which resides on Edgenode. In Oozie I am using a SSH action
to execute the shell script on Edgenode which copies the data to HDFS.

                          I was just wondering if there is any built in API
with in Spark to do this job. I want to read the data from Edgenode into
RDD using JavaSparkContext then do saveAsTextFile("hdfs://...").
JavaSparkContext  does provide any method to pass Edgenode's access
credentials and read the data into an RDD ??

Thank you for your valuable time. Any pointers are appreciated.

Thank You,
Aj

Mime
View raw message