spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: Spark Streaming: Change Kafka topics on runtime
Date Thu, 13 Aug 2015 15:15:18 GMT
The current kafka stream implementation assumes the set of topics doesn't
change during operation.

You could either take a crack at writing a subclass that does what you
need; stop/start; or if your batch duration isn't too small, you could run
it as a series of RDDs (using the existing KafkaUtils.createRDD) where the
set of topics is determined before each rdd.

On Thu, Aug 13, 2015 at 4:38 AM, Nisrina Luthfiyati <
nisrina.luthfiyati@gmail.com> wrote:

> Hi all,
>
> I want to write a Spark Streaming program that listens to Kafka for a list
> of topics.
> The list of topics that I want to consume is stored in a DB and might
> change dynamically. I plan to periodically refresh this list of topics in
> the Spark Streaming app.
>
> My question is is it possible to add/remove a Kafka topic that is consumed
> by a stream, or probably create a new stream at runtime?
> Would I need to stop/start the program or is there any other way to do
> this?
>
> Thanks!
> Nisrina
>

Mime
View raw message