spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Samel <manojsamelt...@gmail.com>
Subject Re: Running make-distribution.sh .. compilation errors in streaming/api/java/JavaPairDStream.scala
Date Wed, 22 Jan 2014 20:14:48 GMT
Did this on a fresh install - still same error.


On Wed, Jan 22, 2014 at 10:46 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> Try doing a sbt clean before rebuilding.
>
> Matei
>
> On Jan 22, 2014, at 10:22 AM, Manoj Samel <manojsameltech@gmail.com>
> wrote:
>
> See thread below. Reposted as compilation error thread
>
> ---------- Forwarded message ----------
> From: Manoj Samel <manojsameltech@gmail.com>
> Date: Wed, Jan 22, 2014 at 10:20 AM
> Subject: Re: make-distribution.sh error
> org.apache.hadoop#hadoop-client;2.0.0: not found
> To: user@spark.incubator.apache.org
>
>
> I think I found the way to go beyond error "unresolved dependency:
> org.apache.hadoop#hadoop-client;2.0.0: not found"
>
> On my machine, the command "hadoop version" gives "2.0.0-cdh4.5.0" , not
> "2.0.0".
>
> After I changed the SPARK_HADOOP_VERSION to 2.0.0-cdh4.5.0, it is
> proceeding beyond that point.
>
> However, now it fails with multiple compilation errors in
> /.../streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
>
> [error]
> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:605:
> type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K, (V,
> com.google.common.base.Optional[W]))]
> [error]  required:
> org.apache.spark.streaming.api.java.JavaPairDStream[K,(V,
> com.google.common.base.Optional[W])]
> [error]  Note: implicit method fromPairDStream is not applicable here
> because it comes after the application point and it lacks an explicit
> result type
> [error]     joinResult.mapValues{case (v, w) => (v,
> JavaUtils.optionToOptional(w))}
> [error]                         ^
> [error]
> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:617:
> type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K,
> (com.google.common.base.Optional[V], W))]
> [error]  required:
> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
> W)]
> [error]  Note: implicit method fromPairDStream is not applicable here
> because it comes after the application point and it lacks an explicit
> result type
> [error]     joinResult.mapValues{case (v, w) =>
> (JavaUtils.optionToOptional(v), w)}
> [error]                         ^
> [error]
> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:632:
> type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K,
> (com.google.common.base.Optional[V], W))]
> [error]  required:
> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
> W)]
> [error]  Note: implicit method fromPairDStream is not applicable here
> because it comes after the application point and it lacks an explicit
> result type
> [error]     joinResult.mapValues{case (v, w) =>
> (JavaUtils.optionToOptional(v), w)}
> [error]                         ^
> [error]
> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:647:
> type mismatch;
> [error]  found   : org.apache.spark.streaming.DStream[(K,
> (com.google.common.base.Optional[V], W))]
> [error]  required:
> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
> W)]
> [error]  Note: implicit method fromPairDStream is not applicable here
> because it comes after the application point and it lacks an explicit
> result type
> [error]     joinResult.mapValues{case (v, w) =>
> (JavaUtils.optionToOptional(v), w)}
> [error]                         ^
> [error] 43 errors found
> [error] (streaming/compile:compile) Compilation failed
>
>
>
>
> On Wed, Jan 22, 2014 at 10:09 AM, Manoj Samel <manojsameltech@gmail.com>wrote:
>
>> Hi,
>>
>> On my cluster I have CDH 4.5 (Hadoop 2.0.0) installed. I installed spark
>> 0.8.1 and tried to run make-distribution.sh after
>> setting SPARK_HADOOP_VERSION=2.0.0
>>
>> I get
>>
>> [warn]
>> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.0.0/hadoop-client-2.0.0.pom
>> [info] Resolving org.apache.derby#derby;10.4.2.0 ...
>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>> [warn] ::          UNRESOLVED DEPENDENCIES         ::
>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>> [warn] :: org.apache.hadoop#hadoop-client;2.0.0: not found
>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>> sbt.ResolveException: unresolved dependency:
>> org.apache.hadoop#hadoop-client;2.0.0: not found
>> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
>>  at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
>> at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
>>  at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
>>
>> Any thoughts?
>>
>
>
>
>

Mime
View raw message