spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: WordCount fails in .textFile() method
Date Wed, 03 Dec 2014 08:55:44 GMT
Try running it in local mode. Looks like a jar conflict/missing.

SparkConf conf = new SparkConf().setAppName("JavaWordCount");
conf.set("spark.io.compression.codec","org.apache.spark.io.LZ4CompressionCodec");
conf.setMaster("*local[2]*").setSparkHome(System.getenv("SPARK_HOME"));
JavaSparkContext jsc = new JavaSparkContext(conf);
jsc.addJar(“myJar.jar”);
new JavaWordCount(jsc).doJob();


Thanks
Best Regards

On Wed, Dec 3, 2014 at 2:49 AM, Rahul Swaminathan <
rahul.swaminathan@duke.edu> wrote:

>  Hi,
>
>  I am trying to run JavaWordCount without using the spark-submit script.
> I have copied the source code for JavaWordCount and am using a
> JavaSparkContext with the following:
>
>  SparkConf conf = new SparkConf().setAppName("JavaWordCount");
>
> conf.set("spark.io.compression.codec","org.apache.spark.io.LZ4CompressionCodec");
> conf.setMaster("spark://127.0.0.1:7077
> ").setSparkHome(System.getenv("SPARK_HOME"));
> JavaSparkContext jsc = new JavaSparkContext(conf);
> jsc.addJar(“myJar.jar”);
>  new JavaWordCount(jsc).doJob();
>
>  I am getting the following error in the .textFile() method:
>
>  Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode;
>
> at org.apache.spark.util.collection.OpenHashSet.org
> $apache$spark$util$collection$OpenHashSet$$hashcode(OpenHashSet.scala:261)
>
>
>  What can I do to solve this issue? It works fine when running from
> command line with spark-submit script.
>
>
>  Thanks,
>
>
>  Rahul
>
>

Mime
View raw message