storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Banias H <banias4sp...@gmail.com>
Subject Help: KafkaSpout not getting data from Kafka
Date Fri, 19 Dec 2014 19:32:54 GMT
Hi folks,

I am new to both Kafka and Storm and I have problem having KafkaSpout to
get data from Kafka in our three-node environment with Kafka 0.8.1.1 and
Storm 0.9.3.

What is working:
- I have a Kafka producer (a java application) to generate random string to
a topic and I was able to run the following command in one of the nodes to
read the random strings on the console while the Kafka producer is running:

<kaffa folder>/bin/kafka-console-consumer.sh --zookeeper
<node1>:2181,<node2>:2181,<node3>:2181/kafka --topic test_topic

- I was also able to run WordCountTopology in Storm.

What is not working:
- I tried running the following code based on KafkaTopology.java:

String zkNodes = "<node1>:2181,<node2>:2181,<node3>:2181";
String brokerZkPath = "/kafka/brokers";
String topicName = "test_topic";
String zkRoot = "/kafka";
String topoName = "test_topology";

ZkHosts zkhost = new ZkHosts(zkNodes, brokerZkPath);
SpoutConfig kafkaConf = new SpoutConfig(zkhost, topicName, zkRoot,
"discovery");
        kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
        kafkaConf.forceStartOffsetTime(-2);
        KafkaSpout kafkaSpout = new KafkaSpout(kafkaConf);

        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("spout", kafkaSpout, 1);
        builder.setBolt("printer", new
PrinterBolt()).shuffleGrouping("spout2");

        Config config = new Config();
        config.setDebug(true);
        config.setNumWorkers(3);
        StormSubmitter.submitTopology(topoName, config,
builder.createTopology());

- Result:
In Storm UI, the numbers of emitted and transferred are always 0,
regardless of whether Kafka producer is running. See attached image. I

- Command to run:
<storm folder>/bin/storm jar storm-starter-0.9.3-jar-with-dependencies.jar
storm.starter.KafkaTopology

- Zookeeper Path:
[zk: <node1>:2181(CONNECTED) 73] ls /kafka
[consumers, config, controller, admin, brokers, controller_epoch]
[zk: <node1>:2181(CONNECTED) 74] ls /kafka/brokers
[topics, ids]

I have run out of ideas of trying different options and figuring out where
to look. If anyone could shed some light on this topic, I would greatly
appreciate it. Many thanks!

-BH

Mime
View raw message