spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Classpath errors with Breeze
Date Mon, 09 Jun 2014 05:03:11 GMT
Hi,

I had a similar problem; I was using `sbt assembly` to build a jar
containing all my dependencies, but since my file system has a problem
with long file names (due to disk encryption), some class files (which
correspond to functions in Scala) where not included in the jar I
uploaded. Although, thinking about it, that would result in a
ClassNotFound exception, not NoSuchMethod. Have you built your code
against a different version of the library than the jar you use in
EC2?

Tobias

On Mon, Jun 9, 2014 at 1:52 PM, dlaw <dieterich.lawson@gmail.com> wrote:
> I'm having some trouble getting a basic matrix multiply to work with Breeze.
> I'm pretty sure it's related to my classpath. My setup is a cluster on AWS
> with 8 m3.xlarges. To create the cluster I used the provided ec2 scripts and
> Spark 1.0.0.
>
> I've made a gist with the relevant pieces of my app:
>
> https://gist.github.com/dieterichlawson/e5e3ab158a09429706e0
>
> The app was created as detailed in the quick start guide.
>
> When I run it I get an error that says the method to multiply a dense matrix
> by a dense matrix does not exist:
>
> 14/06/09 04:49:09 WARN scheduler.TaskSetManager: Lost TID 90 (task 0.0:13)
> 14/06/09 04:49:09 INFO scheduler.TaskSetManager: Loss was due to
> java.lang.NoSuchMethodError:
> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
> [duplicate 46]
>
> I've tried a bunch of different things, including playing with the CLASSPATH
> and ADD_JARS environment variables, the --jars option on spark-submit, the
> version of breeze and scala, etc...
>
> I've also tried it in the spark-shell. It works there, so I don't really
> know what's going on. Any thoughts?
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Classpath-errors-with-Breeze-tp7220.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message