mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: Confusion regarding Samsara's configuration
Date Tue, 02 Feb 2016 17:01:40 GMT
Bahaa, first off, i don't think we have certified any of releases to run
with spar 1.6 (yet). I think spark 1.5 is the last known release to run
with 0.11 series.

Second, if you use mahoutSparkContext() method to create context, it would
look for MAHOUT_HOME setup to add mahout binaries to the job. So the
reasons you may not getting it is perhaps you are not using the
mahoutCreateContext()?

alternatively, you can create context yourself, but you need (1) make sure
it has enabled and configured Kryo serialization properly, and (2) have
added all necessary mahout jars on your own.

-d

On Tue, Feb 2, 2016 at 8:22 AM, BahaaEddin AlAila <bahaelaila7@gmail.com>
wrote:

> Greetings mahout users,
>
> I have been trying to use mahout samsara as a library with scala/spark, but
> I haven't been successful in doing so.
>
> I am running spark 1.6.0 binaries, didn't build it myself.
> However, I tried both readily available binaries on Apache mirrors, and
> cloning and compiling mahout's repo, but neither worked.
>
> I keep getting
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/mahout/sparkbindings/SparkDistributedContext
>
> The way I am doing things is:
> I have spark in ~/spark-1.6
> and mahout in ~/mahout
> I have set both $SPARK_HOME and $MAHOUT_HOME accordingly, along with
> $MAHOUT_LOCAL=true
>
> and I have:
>
> ~/app1/build.sbt
> ~/app1/src/main/scala/App1.scala
>
> in build.sbt I have these lines to declare mahout dependecies:
>
> libraryDependencies += "org.apache.mahout" %% "mahout-math-scala" %
> "0.11.1"
>
> libraryDependencies += "org.apache.mahout" % "mahout-math" % "0.11.1"
>
> libraryDependencies += "org.apache.mahout" % "mahout-spark_2.10" % "0.11.1"
>
> along with other spark dependencies
>
> and in App1.scala, in the main function, I construct a context object using
> mahoutSparkContext, and of course, the sparkbindings are imported
>
> everything compiles successfully
>
> however, when I submit to spark, I get the above mentioned error.
>
> I have a general idea of why this is happening: because the compiled app1
> jar depends on mahout-spark dependency jar but it cannot find it in the
> class path upon being submitted to spark.
>
> In the instructions I couldn't find how to explicitly add the mahout-spark
> dependency jar to the class path.
>
> The question is: Am I doing the configurations correctly or not?
>
> Sorry for the lengthy email
>
> Kind Regards,
> Bahaa
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message