mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: mahout spark FilteredRDD problem
Date Mon, 25 May 2015 18:00:08 GMT
Yes, the master was updated last week. You can check the “commits” tab on github to see
the change history. This puts the master a couple of commits ahead of the release. The master
branch reflects the next release’s work in progress snapshot and currently produces jars
as …0.10.1-SNAPSHOT. Remember not to build for Spark 1.3, leave the POMs as-is. This will
build for 1.2.2 but should run, as far as my testing has shown, on 1.3

In the next week or so there will be a branch that will build and run on Spark 1.3 but will
lack the mahout Scala shell for awhile.


On May 22, 2015, at 8:37 PM, lastarsenal <lastarsenal@163.com> wrote:

this problem is from git clone mahout master, not from mahout 0.10. maybe mahout master
is updated recently? i will update and try again, thank you!

发自我的 iPhone

> 在 2015年5月22日,下午9:20,Pat Ferrel <pat@occamsmachete.com> 写道:
> 
> Mahout 0.10.0 runs on Spark 1.1.1 or below _only_
> 
> If you are only using spark-itemsimilarity you can try the unreleased master, which is
being moved to Spark 1.2.2, and is binary compatible with Spark 1.3. Get the latest master
branch from https://github.com/apache/mahout and build from source. Leave the version at Spark
1.2.2. Changing to Spark 1.3 in the pom will cause compile errors like the ones below.
> 
> I’ve run a few tests on Spark 1.3 but if you’d like to try also please report back
what you find.
> 
> BTW the use of guava has been removed so any source examples are still being updated.
If you are using the command line you should be fine,
> 
> 
> On May 21, 2015, at 11:50 PM, lastarsenal <lastarsenal@163.com> wrote:
> 
> Hi, 
> 
> 
> Recently I tried mahout spark, for example: 
> 
> 
> ./bin/mahout spark-itemsimilarity -i ${input} -o ${output} --master $MyMaster --sparkExecutorMem
2g
> 
> 
> then I met a error like "Caused by: java.lang.ClassCastException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
cannot be cast to org.apache.spark.serializer.KryoRegistrator", It seems that our spark version
is NOT compatible with mahout. 
> 
> 
> For the spark system is deployed by ops, so what I can do just follow them. Then I do
something as below:
> 1.  Modify  <spark.version>1.1.1</spark.version>  to  <spark.version>1.3.0</spark.version>
, which is our spark version,   in pom.xml in mahout.
> 2. Run mvn -DskipTests clean install
> 3. Get a error when build: 
> 
> 
> [ERROR] spark/src/main/scala/org/apache/mahout/sparkbindings/drm/CheckpointedDrmSpark.scala:168:
error: value saveAsSequenceFile is not a member of org.apache.mahout.sparkbindings.DrmRdd[K]
> [ERROR]     rdd.saveAsSequenceFile(path)
> [ERROR]         ^
> [ERROR]spark/src/main/scala/org/apache/mahout/sparkbindings/drm/package.scala:26: error:
object FilteredRDD is not a member of package org.apache.spark.rdd
> [ERROR] import org.apache.spark.rdd.{FilteredRDD, RDD}
> 
> 
> 4. Check 1.3.0 spark, FilteredRDD is dismissed
> 5. Check 1.1.1 spark, FilteredRDD is available.
> 
> 
> So, my question is how can I solve it?
> 
> 
> The error details is as below when I run ./bin/mahout spark-itemsimilarity -i ${input}
-o ${output} --master $MyMaster --sparkExecutorMem 2g
> 15/05/22 12:22:27 WARN TaskSetManager: Lost task 8.0 in stage 0.0 (TID 8, 182.118.21.30):
java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo
>      at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1008)
>      at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
>      at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
>      at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
>      at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
>      at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
>      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:61)
>      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>      at org.apache.spark.scheduler.Task.run(Task.scala:56)
>      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:195)
>      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>      at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.spark.SparkException: Failed to register classes with Kryo
>      at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:105)
>      at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:157)
>      at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:119)
>      at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:214)
>      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:177)
>      at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1005)
>      ... 12 more
> Caused by: java.lang.ClassCastException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
cannot be cast to org.apache.spark.serializer.KryoRegistrator
>      at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$3.apply(KryoSerializer.scala:101)
>      at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$3.apply(KryoSerializer.scala:101)
>      at scala.Option.map(Option.scala:145)
>      at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:101)
>      ... 17 more
> 
> 



Mime
View raw message