spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Spark And Mapr
Date Wed, 01 Oct 2014 23:18:16 GMT
It should just work in PySpark, the same way it does in Java / Scala apps.

Matei

On Oct 1, 2014, at 4:12 PM, Sungwook Yoon <syoon@maprtech.com> wrote:

> 
> Yes.. you should use maprfs://
> 
> I personally haven't used pyspark, I just used scala shell or standalone with MapR.
> 
> I think you need to set classpath right, adding jar like
>  /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar to the classpath
> in the classpath.
> 
> Sungwook
> 
> On Wed, Oct 1, 2014 at 4:09 PM, Addanki, Santosh Kumar <santosh.kumar.addanki@sap.com>
wrote:
> Hi
> 
>  
> 
> We would like to do this in PySpark Environment
> 
>  
> 
> i.e something like
> 
>  
> 
> test = sc.textFile("maprfs:///user/root/test") or
> 
> test = sc.textFile("hdfs:///user/root/test") or
> 
>  
> 
>  
> 
> Currently when we try
> 
> test = sc.textFile("maprfs:///user/root/test")
> 
>  
> 
> It throws the error
> 
> No File-System for scheme: maprfs
> 
>  
> 
>  
> 
> Best Regards
> 
> Santosh
> 
>  
> 
>  
> 
>  
> 
> From: Vladimir Rodionov [mailto:vrodionov@splicemachine.com] 
> Sent: Wednesday, October 01, 2014 3:59 PM
> To: Addanki, Santosh Kumar
> Cc: user@spark.apache.org
> Subject: Re: Spark And Mapr
> 
>  
> 
> There is doc on MapR:
> 
>  
> 
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications
> 
>  
> 
> -Vladimir Rodionov
> 
>  
> 
> On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar <santosh.kumar.addanki@sap.com>
wrote:
> 
> Hi
> 
>  
> 
> We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR
> 
>  
> 
> Previously to read a text file  we would use :
> 
>  
> 
> test = sc.textFile(\"hdfs://10.48.101.111:8020/user/hdfs/test\")"
> 
>  
> 
>  
> 
> What would be the equivalent of the same for Mapr.
> 
>  
> 
> Best Regards
> 
> Santosh
> 
>  
> 
> 


Mime
View raw message