spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Dong <dongda...@gmail.com>
Subject Multiple Kafka topics processing in Spark 2.2
Date Wed, 06 Sep 2017 12:38:38 GMT
Hi, All,
  I have one issue here about how to process multiple Kafka topics in a
Spark 2.* program. My question is: How to get the topic name from a message
received from Kafka? E.g:

......
    val messages = KafkaUtils.createDirectStream[String, String,
StringDecoder, StringDecoder](
      ssc, kafkaParams, topicsSet)

    // Get the lines, split them into words, count the words and print
    val lines = messages.map(_._2)
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1L)).reduceByKey(_ + _)
    wordCounts.print()
......

Kafka send the messages in multiple topics through console producer for
example. But when Spark receive the message, how it will know which topic
is this piece of message coming from? Thanks a lot for any of your helps!

Cheers,
Dan

Mime
View raw message