spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: maprfs and spark libraries
Date Mon, 26 May 2014 14:50:32 GMT
Did you try in standalone mode. You may not see serialization issues in
local threaded mode.
Serialization errors are unlikely to be cause of Mapr hadoop version.
Regards
Mayur

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Mon, May 26, 2014 at 3:18 PM, nelson <nelson.verdier@ysance.com> wrote:

> Hi all,
>
> I am trying to run spark over a MapR cluster. I successfully ran several
> custom applications on a previous non-mapr hadoop cluster but i can't get
> them working on the mapr one. To be more specific, i am not able to read or
> write on mfs without running into a serialization error from Java. Note
> that
> everything works fine when i am running the app in local mode which make me
> think of a dependancy error.
>
> The test application is built using sbt with the following dependency:
>  - org.apache.spark spark-core 0.9.1
>
> In my test_app/lib directory i have:
>  - hadoop-core-1.0.3-mapr-3.0.2.jar
>  - json-20140107.jar
>  - maprfs-1.0.3-mapr-3.0.2.jar
>
> Finally, i add those jars with conf.setJars so that they are distributed on
> the cluster.
>
> Am I compiling with the wrong dependencies? Should i get a "mapr version"
> of
> spark-core?
>
> Regards, Nelson
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message