spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Samel <manojsamelt...@gmail.com>
Subject Re: Running make-distribution.sh .. compilation errors in streaming/api/java/JavaPairDStream.scala
Date Wed, 22 Jan 2014 23:01:36 GMT
Ok, so I found the issue . My initial sbt assembly was not done using
the SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 ...

So I did sbt/sbt clean followed by SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0
sbt/sbt assembly on each of the node (I am not doing the make anymore).
Then I am starting the master and worker nodes. Now I can get proper webUI
with all nodes showing and I get connectivity to HDFS

Thanks Matei for your help





On Wed, Jan 22, 2014 at 2:13 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> Did you do sbt publish-local before by any chance? Maybe it’s picking up
> another build of Spark from somewhere. In that case delete that build from
> your local .m2 and .ivy2 directories.
>
> Matei
>
> On Jan 22, 2014, at 12:14 PM, Manoj Samel <manojsameltech@gmail.com>
> wrote:
>
> Did this on a fresh install - still same error.
>
>
> On Wed, Jan 22, 2014 at 10:46 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>
>> Try doing a sbt clean before rebuilding.
>>
>> Matei
>>
>> On Jan 22, 2014, at 10:22 AM, Manoj Samel <manojsameltech@gmail.com>
>> wrote:
>>
>> See thread below. Reposted as compilation error thread
>>
>> ---------- Forwarded message ----------
>> From: Manoj Samel <manojsameltech@gmail.com>
>> Date: Wed, Jan 22, 2014 at 10:20 AM
>> Subject: Re: make-distribution.sh error
>> org.apache.hadoop#hadoop-client;2.0.0: not found
>> To: user@spark.incubator.apache.org
>>
>>
>> I think I found the way to go beyond error "unresolved dependency:
>> org.apache.hadoop#hadoop-client;2.0.0: not found"
>>
>> On my machine, the command "hadoop version" gives "2.0.0-cdh4.5.0" , not
>> "2.0.0".
>>
>> After I changed the SPARK_HADOOP_VERSION to 2.0.0-cdh4.5.0, it is
>> proceeding beyond that point.
>>
>> However, now it fails with multiple compilation errors in
>> /.../streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
>>
>> [error]
>> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:605:
>> type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K, (V,
>> com.google.common.base.Optional[W]))]
>> [error]  required:
>> org.apache.spark.streaming.api.java.JavaPairDStream[K,(V,
>> com.google.common.base.Optional[W])]
>> [error]  Note: implicit method fromPairDStream is not applicable here
>> because it comes after the application point and it lacks an explicit
>> result type
>> [error]     joinResult.mapValues{case (v, w) => (v,
>> JavaUtils.optionToOptional(w))}
>> [error]                         ^
>> [error]
>> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:617:
>> type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K,
>> (com.google.common.base.Optional[V], W))]
>> [error]  required:
>> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
>> W)]
>> [error]  Note: implicit method fromPairDStream is not applicable here
>> because it comes after the application point and it lacks an explicit
>> result type
>> [error]     joinResult.mapValues{case (v, w) =>
>> (JavaUtils.optionToOptional(v), w)}
>> [error]                         ^
>> [error]
>> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:632:
>> type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K,
>> (com.google.common.base.Optional[V], W))]
>> [error]  required:
>> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
>> W)]
>> [error]  Note: implicit method fromPairDStream is not applicable here
>> because it comes after the application point and it lacks an explicit
>> result type
>> [error]     joinResult.mapValues{case (v, w) =>
>> (JavaUtils.optionToOptional(v), w)}
>> [error]                         ^
>> [error]
>> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:647:
>> type mismatch;
>> [error]  found   : org.apache.spark.streaming.DStream[(K,
>> (com.google.common.base.Optional[V], W))]
>> [error]  required:
>> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
>> W)]
>> [error]  Note: implicit method fromPairDStream is not applicable here
>> because it comes after the application point and it lacks an explicit
>> result type
>> [error]     joinResult.mapValues{case (v, w) =>
>> (JavaUtils.optionToOptional(v), w)}
>> [error]                         ^
>> [error] 43 errors found
>> [error] (streaming/compile:compile) Compilation failed
>>
>>
>>
>>
>> On Wed, Jan 22, 2014 at 10:09 AM, Manoj Samel <manojsameltech@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> On my cluster I have CDH 4.5 (Hadoop 2.0.0) installed. I installed spark
>>> 0.8.1 and tried to run make-distribution.sh after
>>> setting SPARK_HADOOP_VERSION=2.0.0
>>>
>>> I get
>>>
>>> [warn]
>>> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.0.0/hadoop-client-2.0.0.pom
>>> [info] Resolving org.apache.derby#derby;10.4.2.0 ...
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] ::          UNRESOLVED DEPENDENCIES         ::
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: org.apache.hadoop#hadoop-client;2.0.0: not found
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> sbt.ResolveException: unresolved dependency:
>>> org.apache.hadoop#hadoop-client;2.0.0: not found
>>> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
>>>  at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
>>> at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
>>>  at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
>>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
>>>
>>> Any thoughts?
>>>
>>
>>
>>
>>
>
>

Mime
View raw message