storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject Re: Is it possible to choose Kafa version with StormKafka from Storm 1.0.2 ?
Date Thu, 12 Jan 2017 08:48:45 GMT
Hi Alexandre,

README of storm-kafka describes how to change Kafka version.
https://github.com/apache/storm/tree/master/external/storm-kafka#using-storm-kafka-with-different-versions-of-kafka

As README stated, if you use Kafka 0.10.x, please try out new module
storm-kafka-client which is using new client API.

storm-kafka-client 1.0.2 is compatible with Kafka 0.9, and
storm-kafka-client 1.1.0 SNAPSHOT is compatible with Kafka 0.10.x.

I think we're close to start release process of Storm 1.1.0. Before that
you may need to build Storm 1.x branch via mvn install, and include
storm-kafka-client 1.1.0-SNAPSHOT as your fat jar.

Thanks,
Jungtaek Lim

2017년 1월 12일 (목) 오후 12:30, Aaron Niskodé-Dossett <dossett@gmail.com>님이
작성:

Hi Alexandre,

My recollection from trying this when 0.10 was released is that it is
possible, but some small code changes are necessary.  If you're comfortable
building Storm from source, updating the kafka version in the appropriate
poms and compiling would show you what needs to be updated.

On Wed, Jan 11, 2017 at 2:20 PM Alexandre Vermeerbergen <
avermeerbergen@gmail.com> wrote:

Hello,

Any clue on whether or not it's possible to build a KafkaSpout / KafkaBolt
based on Apache Kafka 0.10.x ?

Best regards,
Alexandre


2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:

Hello,

We have been happyly running our real-time supervision homebrewed
application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
& bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.

We're been recently interested in upgrading our setup to Apache Storm
1.0.2, mainly because we sometime hit
https://issues.apache.org/jira/browse/STORM-1879

We also plan to soon upgrade our Kafka Broker cluster to 0.10.x

However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
storm-kafka module

Kafka 0.10 brokers have a new message format which can be consumed and
produced to using older Kafka API, but at the price of performance

Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?

Best regards,
Alexandre

Mime
View raw message