spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: libraryDependencies
Date Wed, 27 Jul 2016 10:57:08 GMT
Hi,

How did you reference "sparksample"? If it ended up in
/Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample I
believe it was referenced as a git-based project in sbt. Is that
correct?

Also, when you "provided" Spark libs you won't be able to run Spark
apps in sbt. See
https://github.com/sbt/sbt-assembly#-provided-configuration. The trick
is to create a test app that executes main of your standalone app.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jul 26, 2016 at 9:18 PM, Martin Somers <sononix@gmail.com> wrote:
>
> my build file looks like
>
> libraryDependencies  ++= Seq(
>               // other dependencies here
>               "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
>               "org.apache.spark" %% "spark-mllib_2.11" % "1.6.0",
>               "org.scalanlp" % "breeze_2.11" % "0.7",
>               // native libraries are not included by default. add this if
> you want them (as of 0.7)
>               // native libraries greatly improve performance, but increase
> jar sizes.
>               "org.scalanlp" % "breeze-natives_2.11" % "0.7",
> )
>
> not 100% sure on the version numbers if they are indeed correct
> getting an error of
>
> [info] Resolving jline#jline;2.12.1 ...
> [info] Done updating.
> [info] Compiling 1 Scala source to
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/target/scala-2.11/classes...
> [error]
> /Users/studio/.sbt/0.13/staging/42f93875138543b4e1d3/sparksample/src/main/scala/MyApp.scala:2:
> object mllib is not a member of package org.apache.spark
> [error] import org.apache.spark.mllib.linalg.distributed.RowMatrix
> ....
> ...
>
>
> Im trying to import in
>
> import org.apache.spark.mllib.linalg.distributed.RowMatrix
> import org.apache.spark.mllib.linalg.SingularValueDecomposition
>
> import org.apache.spark.mllib.linalg.{Vector, Vectors}
>
>
> import breeze.linalg._
> import breeze.linalg.{ Matrix => B_Matrix }
> import breeze.linalg.{ Vector => B_Matrix }
> import breeze.linalg.DenseMatrix
>
> object MyApp {
>   def main(args: Array[String]): Unit = {
> //code here
> }
>
>
> It might not be the correct way of doing this
>
> Anyone got any suggestion
> tks
> M
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message