It should just work in PySpark, the same way it does in Java / Scala apps.

Matei

On Oct 1, 2014, at 4:12 PM, Sungwook Yoon <syoon@maprtech.com> wrote:


Yes.. you should use maprfs://

I personally haven't used pyspark, I just used scala shell or standalone with MapR.

I think you need to set classpath right, adding jar like
 /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar to the classpath
in the classpath.

Sungwook

On Wed, Oct 1, 2014 at 4:09 PM, Addanki, Santosh Kumar <santosh.kumar.addanki@sap.com> wrote:

Hi

 

We would like to do this in PySpark Environment

 

i.e something like

 

test = sc.textFile("maprfs:///user/root/test") or

test = sc.textFile("hdfs:///user/root/test") or

 

 

Currently when we try

test = sc.textFile("maprfs:///user/root/test")

 

It throws the error

No File-System for scheme: maprfs

 

 

Best Regards

Santosh

 

 

 

From: Vladimir Rodionov [mailto:vrodionov@splicemachine.com]
Sent: Wednesday, October 01, 2014 3:59 PM
To: Addanki, Santosh Kumar
Cc: user@spark.apache.org
Subject: Re: Spark And Mapr

 

There is doc on MapR:

 

http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications

 

-Vladimir Rodionov

 

On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar <santosh.kumar.addanki@sap.com> wrote:

Hi

 

We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR

 

Previously to read a text file  we would use :

 

test = sc.textFile(\"hdfs://10.48.101.111:8020/user/hdfs/test\")"

 

 

What would be the equivalent of the same for Mapr.

 

Best Regards

Santosh