spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Fail to use reduceByKey in latest github version with hadoop2
Date Sat, 17 Aug 2013 01:37:14 GMT
Maybe there are some old versions of libraries on the classpath. Run sbt clean and then sbt
package again.

Matei

On Aug 13, 2013, at 11:30 PM, "Huang, Jie" <jie.huang@intel.com> wrote:

> Hi all,
> 
> I am a newbie to Spark. After getting the latest code(0.8.0-SNAPSHOT, the default master
branch) from github, and compiling it with Hadoop2, I found it failed to run the following
example command demonstrated in the Spark document.
>  $> val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word,
1)).reduceByKey((a, b) => a + b)
> 
> BTW, I have prepared the package with SBT, and configured the Hadoop version in SparkBuild.scala
file.  And it doesn’t have the similar problem while choosing hadoop1 option.
> 
> It seems the problem happening with the reduceByKey function(or its class loader). Since
I have tried those RDD transformations one by one.  And you can find more details as below.
> Is there any solution to this problem? Is there anything missing in my preparation work?
Thank you very much for your time and help.
> 
> ---------------------------------------------------------------------------------------------------------------------------
> java.lang.IncompatibleClassChangeError: Implementing class
>        at java.lang.ClassLoader.defineClass1(Native Method)
>        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:88)
>        at scala.tools.nsc.util.ScalaClassLoader$class.findClass(ScalaClassLoader.scala:44)
>        at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.findClass(ScalaClassLoader.scala:88)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$loadClass(ScalaClassLoader.scala:88)
>        at scala.tools.nsc.util.ScalaClassLoader$class.loadClass(ScalaClassLoader.scala:50)
>        at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.loadClass(ScalaClassLoader.scala:88)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        at spark.PairRDDFunctions.<init>(PairRDDFunctions.scala:53)
>        at spark.SparkContext$.rddToPairRDDFunctions(SparkContext.scala:877)
>        at <init>(<console>:14)
>        at <init>(<console>:19)
>        at <init>(<console>:21)
>        at <init>(<console>:23)
>        at <init>(<console>:25)
>        at .<init>(<console>:29)
>        at .<clinit>(<console>)
>        at .<init>(<console>:11)
>        at .<clinit>(<console>)
>        at $export(<console>)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)
>        at spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:890)
>        at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
>        at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
>        at java.lang.Thread.run(Thread.java:662)
> 
> Thank you && Best Regards,
> Grace (Huang Jie)
> 
> 


Mime
View raw message