You must be doing *something* wrong.

One thing that stands out to me is that you're using a different version of the kafka-client (0.10.1.1) than storm-kafka-client has configured (0.9.0.1):
If the problem isn't related to that, then I'd say you are somehow not including kafka-clients jar into your uber jar.  Can you "jar tf" your uber jar and ensure it has the KafkaConsumer class in it?  Then you could use "jar xf" to get the .class file, and then use "javap" to decompile and look for the KafkaConsumer.subscribe() method.

- Erik

On Sat, Feb 25, 2017 at 1:26 PM, pradeep s <sreekumar.pradeep@gmail.com> wrote:
Hi,
Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as in storm lib. Still its failing with same error.
KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded from storm-kafka-client-1.0.3.jar.

KafkaSpout source code shows below. Its passing a List of topics to subscribe call.

private void subscribeKafkaConsumer() {

        kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig.getKafkaProps(),

                kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig.getValueDeserializer());


        if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {

            final List<String> topics = ((KafkaSpoutStreamsNamedTopics) kafkaSpoutStreams).getTopics();

            kafkaConsumer.subscribe(topics, new KafkaSpoutConsumerRebalanceListener());

            LOG.info("Kafka consumer subscribed topics {}", topics);

        } 


Verified the worker logs and confirmed that its loading the proper kafka version.

2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka version : 0.10.1.1

2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka commitId : f10ef2720b03b247

2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka version : 0.10.1.1

2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka commitId : f10ef2720b03b247

2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1

2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId : f10ef2720b03b247

2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka version : 0.10.1.1

2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka commitId : f10ef2720b03b247



Regards

Pradeep S


On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <kunal.gupta@kartrocket.com> wrote:
Please add Kafka maven repo in Storm lib directory

On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sreekumar.pradeep@gmail.com> wrote:
Hi,
I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout using storm-kafka-client
Dependencies used are 

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka-client</artifactId>
<version>1.0.3</version>
</dependency>

While starting the topology spout is failing by below error
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
        at org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360) ~[stormjar.jar:?]
        at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351) ~[stormjar.jar:?]
        at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639) ~[storm-core-1.0.3.jar:1.0.3]
        at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) [storm-core-1.0.3.jar:1.0.3]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

Regards
Pradeep S



--
Thanks Kunal
Senior Software Engineer
Bigfoot Retail Solutions Pvt Ltd.
Khasra No. 264, Westend Marg, Saidulajab, Saket 110030

http://www.kartrocket.com