spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nelson <nelson.verd...@ysance.com>
Subject maprfs and spark libraries
Date Mon, 26 May 2014 09:48:10 GMT
Hi all,

I am trying to run spark over a MapR cluster. I successfully ran several
custom applications on a previous non-mapr hadoop cluster but i can't get
them working on the mapr one. To be more specific, i am not able to read or
write on mfs without running into a serialization error from Java. Note that
everything works fine when i am running the app in local mode which make me
think of a dependancy error.

The test application is built using sbt with the following dependency:
 - org.apache.spark spark-core 0.9.1

In my test_app/lib directory i have:
 - hadoop-core-1.0.3-mapr-3.0.2.jar
 - json-20140107.jar
 - maprfs-1.0.3-mapr-3.0.2.jar

Finally, i add those jars with conf.setJars so that they are distributed on
the cluster.

Am I compiling with the wrong dependencies? Should i get a "mapr version" of
spark-core?

Regards, Nelson



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message