spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Running make-distribution.sh .. compilation errors in streaming/api/java/JavaPairDStream.scala
Date Wed, 22 Jan 2014 22:13:12 GMT
Did you do sbt publish-local before by any chance? Maybe it’s picking up another build of
Spark from somewhere. In that case delete that build from your local .m2 and .ivy2 directories.

Matei

On Jan 22, 2014, at 12:14 PM, Manoj Samel <manojsameltech@gmail.com> wrote:

> Did this on a fresh install - still same error.
> 
> 
> On Wed, Jan 22, 2014 at 10:46 AM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
> Try doing a sbt clean before rebuilding.
> 
> Matei
> 
> On Jan 22, 2014, at 10:22 AM, Manoj Samel <manojsameltech@gmail.com> wrote:
> 
>> See thread below. Reposted as compilation error thread
>> 
>> ---------- Forwarded message ----------
>> From: Manoj Samel <manojsameltech@gmail.com>
>> Date: Wed, Jan 22, 2014 at 10:20 AM
>> Subject: Re: make-distribution.sh error org.apache.hadoop#hadoop-client;2.0.0: not
found
>> To: user@spark.incubator.apache.org
>> 
>> 
>> I think I found the way to go beyond error "unresolved dependency: org.apache.hadoop#hadoop-client;2.0.0:
not found"
>> 
>> On my machine, the command "hadoop version" gives "2.0.0-cdh4.5.0" , not "2.0.0".
>> 
>> After I changed the SPARK_HADOOP_VERSION to 2.0.0-cdh4.5.0, it is proceeding beyond
that point.
>> 
>> However, now it fails with multiple compilation errors in /.../streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
>> 
>> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:605:
type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K, (V, com.google.common.base.Optional[W]))]
>> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(V, com.google.common.base.Optional[W])]
>> [error]  Note: implicit method fromPairDStream is not applicable here because it
comes after the application point and it lacks an explicit result type
>> [error]     joinResult.mapValues{case (v, w) => (v, JavaUtils.optionToOptional(w))}
>> [error]                         ^
>> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:617:
type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K, (com.google.common.base.Optional[V],
W))]
>> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
>> [error]  Note: implicit method fromPairDStream is not applicable here because it
comes after the application point and it lacks an explicit result type
>> [error]     joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v),
w)}
>> [error]                         ^
>> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:632:
type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K, (com.google.common.base.Optional[V],
W))]
>> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
>> [error]  Note: implicit method fromPairDStream is not applicable here because it
comes after the application point and it lacks an explicit result type
>> [error]     joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v),
w)}
>> [error]                         ^
>> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:647:
type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K, (com.google.common.base.Optional[V],
W))]
>> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
>> [error]  Note: implicit method fromPairDStream is not applicable here because it
comes after the application point and it lacks an explicit result type
>> [error]     joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v),
w)}
>> [error]                         ^
>> [error] 43 errors found
>> [error] (streaming/compile:compile) Compilation failed
>> 
>> 
>> 
>> 
>> On Wed, Jan 22, 2014 at 10:09 AM, Manoj Samel <manojsameltech@gmail.com> wrote:
>> Hi,
>> 
>> On my cluster I have CDH 4.5 (Hadoop 2.0.0) installed. I installed spark 0.8.1 and
tried to run make-distribution.sh after setting SPARK_HADOOP_VERSION=2.0.0
>> 
>> I get 
>> 
>> [warn]   http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.0.0/hadoop-client-2.0.0.pom
>> [info] Resolving org.apache.derby#derby;10.4.2.0 ...
>> [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
>> [warn] 	::          UNRESOLVED DEPENDENCIES         ::
>> [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
>> [warn] 	:: org.apache.hadoop#hadoop-client;2.0.0: not found
>> [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
>> sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-client;2.0.0:
not found
>> 	at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
>> 	at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
>> 	at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
>> 	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
>> 	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
>> 
>> Any thoughts?
>> 
>> 
> 
> 


Mime
View raw message