storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Weathers <eweath...@groupon.com>
Subject Re: Storm kafka spout no such method error
Date Sun, 26 Feb 2017 00:21:14 GMT
You must be doing *something* wrong.

One thing that stands out to me is that you're using a different version of
the kafka-client (0.10.1.1) than storm-kafka-client has configured
(0.9.0.1):

   - https://github.com/apache/storm/blob/v1.0.3/pom.xml#L256-L257

If the problem isn't related to that, then I'd say you are somehow not
including kafka-clients jar into your uber jar.  Can you "jar tf" your uber
jar and ensure it has the KafkaConsumer class in it?  Then you could use
"jar xf" to get the .class file, and then use "javap" to decompile and look
for the KafkaConsumer.subscribe() method.

- Erik

On Sat, Feb 25, 2017 at 1:26 PM, pradeep s <sreekumar.pradeep@gmail.com>
wrote:

> Hi,
> Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as
> in storm lib. Still its failing with same error.
> KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
> KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded
> from storm-kafka-client-1.0.3.jar.
>
> *KafkaSpout source code shows below*. Its passing a List of topics to
> subscribe call.
>
> private void subscribeKafkaConsumer() {
>
>         kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig
> .getKafkaProps(),
>
>                 kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig.
> getValueDeserializer());
>
>
>         if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {
>
>       *      final List<String> topics = ((KafkaSpoutStreamsNamedTopics)
> kafkaSpoutStreams).getTopics();*
>
>             kafkaConsumer.subscribe(topics, new
> KafkaSpoutConsumerRebalanceListener());
>
>             LOG.info("Kafka consumer subscribed topics {}", topics);
>
>         }
>
>
> Verified the worker logs and confirmed that its loading the proper kafka
> version.
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser Thread-11-merchMariaBolt-executor[102
> 103] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser Thread-11-merchMariaBolt-executor[102
> 103] [INFO] Kafka commitId : f10ef2720b03b247
>
> 2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser Thread-9-merchMariaBolt-executor[92
> 93] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser Thread-9-merchMariaBolt-executor[92
> 93] [INFO] Kafka commitId : f10ef2720b03b247
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser Thread-65-merchMariaBolt-executor[32
> 33] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser Thread-65-merchMariaBolt-executor[32
> 33] [INFO] Kafka commitId : f10ef2720b03b247
>
>
>
> Regards
>
> Pradeep S
>
> On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <kunal.gupta@kartrocket.com>
> wrote:
>
>> Please add Kafka maven repo in Storm lib directory
>>
>> On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sreekumar.pradeep@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
>>> using storm-kafka-client
>>> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
>>> Dependencies used are
>>>
>>> <dependency>
>>> <groupId>org.apache.kafka</groupId>
>>> <artifactId>kafka-clients</artifactId>
>>> <version>0.10.1.1</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.storm</groupId>
>>> <artifactId>storm-kafka-client</artifactId>
>>> <version>1.0.3</version>
>>> </dependency>
>>>
>>> While starting the topology spout is failing by below error
>>> java.lang.*NoSuchMethodError*: org.apache.kafka.clients.consu
>>> mer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafk
>>> a/clients/consumer/ConsumerRebalanceListener;)V
>>>         at org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
>>> ~[stormjar.jar:?]
>>>         at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
>>> ~[stormjar.jar:?]
>>>         at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
>>> ~[storm-core-1.0.3.jar:1.0.3]
>>>         at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
>>> [storm-core-1.0.3.jar:1.0.3]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>>>
>>> Regards
>>> Pradeep S
>>>
>>
>>
>>
>> --
>> *Thanks **Kunal*
>> *+91-9958189589 <+91%2099581%2089589>*
>> *Senior Software Engineer*
>>
>>
>> *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg,
>> Saidulajab, Saket 110030http://www.kartrocket.com
>> <http://www.kartrocket.com/>*
>>
>
>

Mime
View raw message