spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Running make-distribution.sh .. compilation errors in streaming/api/java/JavaPairDStream.scala
Date Wed, 22 Jan 2014 18:46:18 GMT
Try doing a sbt clean before rebuilding.

Matei

On Jan 22, 2014, at 10:22 AM, Manoj Samel <manojsameltech@gmail.com> wrote:

> See thread below. Reposted as compilation error thread
> 
> ---------- Forwarded message ----------
> From: Manoj Samel <manojsameltech@gmail.com>
> Date: Wed, Jan 22, 2014 at 10:20 AM
> Subject: Re: make-distribution.sh error org.apache.hadoop#hadoop-client;2.0.0: not found
> To: user@spark.incubator.apache.org
> 
> 
> I think I found the way to go beyond error "unresolved dependency: org.apache.hadoop#hadoop-client;2.0.0:
not found"
> 
> On my machine, the command "hadoop version" gives "2.0.0-cdh4.5.0" , not "2.0.0".
> 
> After I changed the SPARK_HADOOP_VERSION to 2.0.0-cdh4.5.0, it is proceeding beyond that
point.
> 
> However, now it fails with multiple compilation errors in /.../streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
> 
> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:605:
type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K, (V, com.google.common.base.Optional[W]))]
> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(V, com.google.common.base.Optional[W])]
> [error]  Note: implicit method fromPairDStream is not applicable here because it comes
after the application point and it lacks an explicit result type
> [error]     joinResult.mapValues{case (v, w) => (v, JavaUtils.optionToOptional(w))}
> [error]                         ^
> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:617:
type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K, (com.google.common.base.Optional[V],
W))]
> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
> [error]  Note: implicit method fromPairDStream is not applicable here because it comes
after the application point and it lacks an explicit result type
> [error]     joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v), w)}
> [error]                         ^
> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:632:
type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K, (com.google.common.base.Optional[V],
W))]
> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
> [error]  Note: implicit method fromPairDStream is not applicable here because it comes
after the application point and it lacks an explicit result type
> [error]     joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v), w)}
> [error]                         ^
> [error] /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:647:
type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K, (com.google.common.base.Optional[V],
W))]
> [error]  required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
> [error]  Note: implicit method fromPairDStream is not applicable here because it comes
after the application point and it lacks an explicit result type
> [error]     joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v), w)}
> [error]                         ^
> [error] 43 errors found
> [error] (streaming/compile:compile) Compilation failed
> 
> 
> 
> 
> On Wed, Jan 22, 2014 at 10:09 AM, Manoj Samel <manojsameltech@gmail.com> wrote:
> Hi,
> 
> On my cluster I have CDH 4.5 (Hadoop 2.0.0) installed. I installed spark 0.8.1 and tried
to run make-distribution.sh after setting SPARK_HADOOP_VERSION=2.0.0
> 
> I get 
> 
> [warn]   http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.0.0/hadoop-client-2.0.0.pom
> [info] Resolving org.apache.derby#derby;10.4.2.0 ...
> [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
> [warn] 	::          UNRESOLVED DEPENDENCIES         ::
> [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
> [warn] 	:: org.apache.hadoop#hadoop-client;2.0.0: not found
> [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
> sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-client;2.0.0: not
found
> 	at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
> 	at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
> 	at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
> 	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
> 	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
> 
> Any thoughts?
> 
> 


Mime
View raw message