spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: maven doesn't build dependencies with Scala 2.11
Date Mon, 19 Jan 2015 03:36:04 GMT
bq. there was no 2.11 Kafka available

That's right. Adding external/kafka module resulted in:

[ERROR] Failed to execute goal on project spark-streaming-kafka_2.11: Could
not resolve dependencies for project
org.apache.spark:spark-streaming-kafka_2.11:jar:1.3.0-SNAPSHOT: Could not
find artifact org.apache.kafka:kafka_2.11:jar:0.8.0 in central (
https://repo1.maven.org/maven2) -> [Help 1]

Cheers

On Sun, Jan 18, 2015 at 10:41 AM, Sean Owen <sowen@cloudera.com> wrote:

> I could be wrong, but I thought this was on purpose. At the time it
> was set up, there was no 2.11 Kafka available? or one of its
> dependencies wouldn't work with 2.11?
>
> But I'm not sure what the OP means by "maven doesn't build Spark's
> dependencies" because Ted indicates it does, and of course you can see
> that these artifacts are published.
>
> On Sun, Jan 18, 2015 at 2:46 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > There're 3 jars under lib_managed/jars directory with and without
> > -Dscala-2.11 flag.
> >
> > Difference between scala-2.10 and scala-2.11 profiles is that scala-2.10
> > profile has the following:
> >       <modules>
> >         <module>external/kafka</module>
> >       </modules>
> >
> > FYI
> >
> > On Sat, Jan 17, 2015 at 4:07 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >> I did the following:
> >>  1655  dev/change-version-to-2.11.sh
> >>  1657  mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
> >> -Dscala-2.11 -DskipTests clean package
> >>
> >> And mvn command passed.
> >>
> >> Did you see any cross-compilation errors ?
> >>
> >> Cheers
> >>
> >> BTW the two links you mentioned are consistent in terms of building for
> >> Scala 2.11
> >>
> >> On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <walrusthecat@gmail.com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> When I run this:
> >>>
> >>> dev/change-version-to-2.11.sh
> >>> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
> >>>
> >>> as per here, maven doesn't build Spark's dependencies.
> >>>
> >>> Only when I run:
> >>>
> >>> dev/change-version-to-2.11.sh
> >>> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests  clean package
> >>>
> >>> as gathered from here, do I get Spark's dependencies built without any
> >>> cross-compilation errors.
> >>>
> >>> Question:
> >>>
> >>> - How can I make maven do this?
> >>>
> >>> - How can I specify the use of Scala 2.11 in my own .pom files?
> >>>
> >>> Thanks
> >>
> >>
> >
>

Mime
View raw message