spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: version mismatch issue with spark breeze vector
Date Thu, 23 Oct 2014 00:52:22 GMT
Hi Yang,

It looks like your build file is a different version than the version of
Spark you are running against. I'd try building against the same version of
spark as you are running your application against (1.1.0). Also what is
your assembly/shading configuration for your build?

Cheers,

Holden :)

On Wed, Oct 22, 2014 at 5:39 PM, Yang <teddyyyy123@gmail.com> wrote:

>
> I'm trying to submit a simple test code through spark-submit. first
> portion of the code works fine, but some calls to breeze vector library
> fails:
>
> 14/10/22 17:36:02 INFO CacheManager: Partition rdd_1_0 not found,
> computing it
> 14/10/22 17:36:02 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
> 0)
> java.lang.NoSuchMethodError:
> breeze.linalg.SparseVector$mcD$sp.<init>([I[DILbreeze/storage/DefaultArrayValue;)V
> at spark.kmeans.SparseVector.toBreeze(Vectors.scala:182)
> at spark.kmeans.MyTest$$anonfun$1.apply(MyTest.scala:56)
> at spark.kmeans.MyTest$$anonfun$1.apply(MyTest.scala:56)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:236)
>
>
> I ran this code with
> ~/tools/spark-1.1.0-bin-hadoop2.4/bin/spark-submit  --class
> spark.kmeans.MyTest --master local target/reco-v1-jar-with-dependencies.jar
>
>
>
> my system scala version is 2.10.4
>
> the relevant sections in my pom is: (I did not build with sbt, since it
> gave me some problems)
>                 <dependency>
>                         <groupId>org.apache.spark</groupId>
>                         <artifactId>spark-core_2.10</artifactId>
>                         <version>1.0.2</version>
>                         <exclusions>
>                                 <exclusion>
>
> <groupId>org.apache.hadoop</groupId>
>
> <artifactId>hadoop-client</artifactId>
>                                 </exclusion>
>                         </exclusions>
>                 </dependency>
>
>                 <dependency>
>                     <groupId>org.apache.spark</groupId>
>                     <artifactId>spark-mllib_2.10</artifactId>
>                     <version>1.0.2</version>
>                 </dependency>
>
>              <dependency>
>             <groupId>org.scala-lang</groupId>
>             <artifactId>scala-library</artifactId>
>             <version>2.10.4</version>
>         </dependency>
>
>
>
> Thanks a lot
> Yang
>



-- 
Cell : 425-233-8271

Mime
View raw message