spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeffrey Jedele <jeffrey.jed...@gmail.com>
Subject Re: Spark Streaming Not Reading Messages From Multiple Kafka Topics
Date Sat, 21 Mar 2015 10:08:07 GMT
Hey Eason!
Weird problem indeed. More information will probably help to find te issue:

Have you searched the logs for peculiar messages?
How does your Spark environment look like? #workers, #threads, etc?
Does it work if you create separate receivers for the topics?

Regards,
Jeff

2015-03-21 2:27 GMT+01:00 EH <easonh@gmail.com>:

> Hi all,
>
> I'm building a Spark Streaming application that will continuously read
> multiple kafka topics at the same time.  However, I found a weird issue
> that
> it reads only hundreds of messages then it stopped reading any more.  If I
> changed the three topic to only one topic, then it is fine and it will
> continue to consume.  Below is the code I have.
>
> val consumerThreadsPerInputDstream = 1
> val topics = Map("raw_0" -> consumerThreadsPerInputDstream)
>                          "raw_1" -> consumerThreadsPerInputDstream,
>                          "raw_2" -> consumerThreadsPerInputDstream)
>
> val msgs = KafkaUtils.createStream(ssc, "10.10.10.10:2181/hkafka",
> "group01", topics).map(_._2)
> .......
>
> How come it will no longer consume after hundreds of messages for three
> topic reading?  How to resolve this issue?
>
> Thank you for your help,
> Eason
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Not-Reading-Messages-From-Multiple-Kafka-Topics-tp22170.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message