spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vibhakar, Beejal" <Beejal.Vibha...@fisglobal.com>
Subject Consuming Data in Parallel using Spark Streaming
Date Thu, 22 Feb 2018 03:12:45 GMT
I am trying to process data from 3 different Kafka topics using 3 InputDStream with a single
StreamingContext. I am currently testing this under Sandbox where I see data processed from
one Kafka topic followed by other.

Question#1: I want to understand that when I run this program in Hadoop cluster, will it process
the data in parallel from 3 Kafka topics OR will I see the same behavior as I see in my Sandbox?

Question#2: I aim to process the data from all three Kafka topics in parallel.  Can I achieve
this without breaking this program into 3 separate smaller programs?

Here's how the code template looks like..

       val ssc = new StreamingContext(sc, 30)

val topic1 = Array("TOPIC1")

       val dataStreamTopic1 = KafkaUtils.createDirectStream[Array[Byte], GenericRecord](
      ssc,
      PreferConsistent,
      Subscribe[Array[Byte], GenericRecord](topic1, kafkaParms))

             // Processing logic for dataStreamTopic1


val topic2 = Array("TOPIC2")

       val dataStreamTopic2 = KafkaUtils.createDirectStream[Array[Byte], GenericRecord](
      ssc,
      PreferConsistent,
      Subscribe[Array[Byte], GenericRecord](topic2, kafkaParms))

             // Processing logic for dataStreamTopic2


val topic3 = Array("TOPIC3")

       val dataStreamTopic3 = KafkaUtils.createDirectStream[Array[Byte], GenericRecord](
      ssc,
      PreferConsistent,
      Subscribe[Array[Byte], GenericRecord](topic3, kafkaParms))

             // Processing logic for dataStreamTopic3

    // Start the Streaming
    ssc.start()
    ssc.awaitTermination()

Here's how I submit my spark job on my sandbox...

./bin/spark-submit --class <CLASS NAME> --master local[*] <PATH TO JAR>

Thanks,
Beejal


The information contained in this message is proprietary and/or confidential. If you are not
the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review
by persons other than the intended recipient. Thank you.

Mime
View raw message