spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <>
Subject Spark Streaming action not triggered with Kafka inputs
Date Fri, 23 Jan 2015 19:25:51 GMT
I am running into some problems with Spark Streaming when reading from
Kafka.I used Spark 1.2.0 built on CDH5.
The example is based on:
* It works with default implementation.
val topicMap = topics.split(",").map((_,numThreads.toInt)).toMap
val lines = KafkaUtils.createStream(ssc, zkQuorum, group,

* However, when I changed it to parallel receiving, like shown below

val topicMap = topics.split(",").map((_, 1)).toMap
val parallelInputs = (1 to numThreads.toInt) map { _ =>         KafkaUtils.
createStream(ssc, zkQuorum, group, topicMap)


After the change, the job stage just hang there and never finish. It looks
like no action is triggered on the streaming job. When I check the
"Streaming" tab, it show messages below:
Batch Processing Statistics

   No statistics have been generated yet.

Am I doing anything wrong on the parallel receiving part?

Chen Song

View raw message