storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject Re: Storm kafka spout no such method error
Date Sun, 26 Feb 2017 11:15:14 GMT
Unfortunately, storm-kafka-client 1.0.x is compatible with Kafka 0.9 and
storm-kafka-client 1.1.0 (not released yet but we'll make it happen) is
compatible with Kafka 0.10.

https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/README.md
https://github.com/apache/storm/blob/1.x-branch/docs/storm-kafka-client.md

You may want to participate release vote for 1.1.0 RC2 and make it happen
soon.

Thanks,
Jungtaek Lim (HeartSaVioR)


2017년 2월 26일 (일) 오전 9:21, Erik Weathers <eweathers@groupon.com>님이 작성:

> You must be doing *something* wrong.
>
> One thing that stands out to me is that you're using a different version
> of the kafka-client (0.10.1.1) than storm-kafka-client has configured
> (0.9.0.1):
>
>    - https://github.com/apache/storm/blob/v1.0.3/pom.xml#L256-L257
>
> If the problem isn't related to that, then I'd say you are somehow not
> including kafka-clients jar into your uber jar.  Can you "jar tf" your uber
> jar and ensure it has the KafkaConsumer class in it?  Then you could use
> "jar xf" to get the .class file, and then use "javap" to decompile and look
> for the KafkaConsumer.subscribe() method.
>
> - Erik
>
> On Sat, Feb 25, 2017 at 1:26 PM, pradeep s <sreekumar.pradeep@gmail.com>
> wrote:
>
> Hi,
> Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as
> in storm lib. Still its failing with same error.
> KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
> KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded
> from storm-kafka-client-1.0.3.jar.
>
> *KafkaSpout source code shows below*. Its passing a List of topics to
> subscribe call.
>
> private void subscribeKafkaConsumer() {
>
>         kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig
> .getKafkaProps(),
>
>                 kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig
> .getValueDeserializer());
>
>
>         if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {
>
>       *      final List<String> topics = ((KafkaSpoutStreamsNamedTopics)
> kafkaSpoutStreams).getTopics();*
>
>             kafkaConsumer.subscribe(topics, new
> KafkaSpoutConsumerRebalanceListener());
>
>             LOG.info("Kafka consumer subscribed topics {}", topics);
>
>         }
>
>
> Verified the worker logs and confirmed that its loading the proper kafka
> version.
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
> Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
> Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser
> Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser
> Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser
> Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka commitId :
> f10ef2720b03b247
>
>
>
> Regards
>
> Pradeep S
>
> On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <kunal.gupta@kartrocket.com>
> wrote:
>
> Please add Kafka maven repo in Storm lib directory
>
> On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sreekumar.pradeep@gmail.com>
> wrote:
>
> Hi,
> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
> using storm-kafka-client
> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
> Dependencies used are
>
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka-clients</artifactId>
> <version>0.10.1.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.storm</groupId>
> <artifactId>storm-kafka-client</artifactId>
> <version>1.0.3</version>
> </dependency>
>
> While starting the topology spout is failing by below error
> java.lang.*NoSuchMethodError*:
> org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
>         at
> org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
> ~[stormjar.jar:?]
>         at
> org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
> ~[stormjar.jar:?]
>         at
> org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
> ~[storm-core-1.0.3.jar:1.0.3]
>         at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.0.3.jar:1.0.3]
>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>
> Regards
> Pradeep S
>
>
>
>
> --
> *Thanks **Kunal*
> *+91-9958189589 <+91%2099581%2089589>*
> *Senior Software Engineer*
>
>
> *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg,
> Saidulajab, Saket 110030http://www.kartrocket.com
> <http://www.kartrocket.com/>*
>
>
>
>

Mime
View raw message