spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tapas Swain <tapas.030...@gmail.com>
Subject Only One Kafka receiver is running in spark irrespective of multiple DStreams
Date Thu, 01 Jan 2015 09:59:53 GMT
Hi All,

I am consuming a 8 partition kafka topic through multiple Dstreams and
Processing them in Spark.
But irrespective of multiple InputDstreams the spark master UI is showing
only one receiver.

The following is the consumer part of spark code:
int numStreams = 8;
		List<JavaPairDStream&lt;String, String>> kafkaStreams = new
ArrayList<JavaPairDStream&lt;String, String>>(
				numStreams);
		for (int i = 0; i < numStreams; i++) {
			kafkaStreams.add(KafkaUtils.createStream(jssc,
					"bhucloud04.adxyz.com:2181", "test-consumer-group",
					topicMap));
		}
		JavaPairDStream<String, String> unifiedStream = jssc.union(
				kafkaStreams.get(0),
				kafkaStreams.subList(1, kafkaStreams.size()));
		unifiedStream.repartition(8);


I have attached the screen shot of 
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n20934/spark-kafka.png>
spark ui 


Thanks
Tapas



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Only-One-Kafka-receiver-is-running-in-spark-irrespective-of-multiple-DStreams-tp20934.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message