bq. there was no 2.11 Kafka available

That's right. Adding external/kafka module resulted in:

[ERROR] Failed to execute goal on project spark-streaming-kafka_2.11: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka_2.11:jar:1.3.0-SNAPSHOT: Could not find artifact org.apache.kafka:kafka_2.11:jar:0.8.0 in central (https://repo1.maven.org/maven2) -> [Help 1]

Cheers

On Sun, Jan 18, 2015 at 10:41 AM, Sean Owen <sowen@cloudera.com> wrote:
I could be wrong, but I thought this was on purpose. At the time it
was set up, there was no 2.11 Kafka available? or one of its
dependencies wouldn't work with 2.11?

But I'm not sure what the OP means by "maven doesn't build Spark's
dependencies" because Ted indicates it does, and of course you can see
that these artifacts are published.

On Sun, Jan 18, 2015 at 2:46 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> There're 3 jars under lib_managed/jars directory with and without
> -Dscala-2.11 flag.
>
> Difference between scala-2.10 and scala-2.11 profiles is that scala-2.10
> profile has the following:
>       <modules>
>         <module>external/kafka</module>
>       </modules>
>
> FYI
>
> On Sat, Jan 17, 2015 at 4:07 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> I did the following:
>>  1655  dev/change-version-to-2.11.sh
>>  1657  mvn -DHADOOP_PROFILE=hadoop-2.4 -Pyarn,hive -Phadoop-2.4
>> -Dscala-2.11 -DskipTests clean package
>>
>> And mvn command passed.
>>
>> Did you see any cross-compilation errors ?
>>
>> Cheers
>>
>> BTW the two links you mentioned are consistent in terms of building for
>> Scala 2.11
>>
>> On Sat, Jan 17, 2015 at 3:43 PM, Walrus theCat <walrusthecat@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> When I run this:
>>>
>>> dev/change-version-to-2.11.sh
>>> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
>>>
>>> as per here, maven doesn't build Spark's dependencies.
>>>
>>> Only when I run:
>>>
>>> dev/change-version-to-2.11.sh
>>> sbt/sbt -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests  clean package
>>>
>>> as gathered from here, do I get Spark's dependencies built without any
>>> cross-compilation errors.
>>>
>>> Question:
>>>
>>> - How can I make maven do this?
>>>
>>> - How can I specify the use of Scala 2.11 in my own .pom files?
>>>
>>> Thanks
>>
>>
>