spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean McNamara <Sean.McNam...@Webtrends.com>
Subject Re: Kafka version dependency in Spark 1.2
Date Mon, 10 Nov 2014 19:11:24 GMT
> Can the 0.8.1.1 client still talk to 0.8.0 versions of Kafka

Yes it can.

"0.8.1 is fully compatible with 0.8."  It is buried on this page: http://kafka.apache.org/documentation.html

In addition to the pom version bump SPARK-2492 would bring the kafka streaming receiver (which
was originally based on kafka 0.7)  in line with kafka 0.8:
https://issues.apache.org/jira/browse/SPARK-2492
https://github.com/apache/spark/pull/1420

I will soon test that PR on a spark+yarn cluster

Cheers,

Sean


On Nov 10, 2014, at 11:58 AM, Matei Zaharia <matei.zaharia@gmail.com<mailto:matei.zaharia@gmail.com>>
wrote:

Just curious, what are the pros and cons of this? Can the 0.8.1.1 client still talk to 0.8.0
versions of Kafka, or do you need it to match your Kafka version exactly?

Matei

On Nov 10, 2014, at 9:48 AM, Bhaskar Dutta <bhaskie@gmail.com<mailto:bhaskie@gmail.com>>
wrote:

Hi,

Is there any plan to bump the Kafka version dependency in Spark 1.2 from 0.8.0 to 0.8.1.1?

Current dependency is still on Kafka 0.8.0
https://github.com/apache/spark/blob/branch-1.2/external/kafka/pom.xml

Thanks
Bhaskie



Mime
View raw message