spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudarshan Kadambi (BLOOMBERG/ 731 LEX)" <>
Subject Re:Adding Kafka topics to a running streaming context
Date Thu, 27 Aug 2015 14:24:53 GMT
This is something we have been needing for a while too. We are restarting the streaming context
to handle new topic subscriptions & unsubscriptions which affects latency of update handling.
I think this is something that needs to be addressed in core Spark Streaming (I can't think
of any fundamental limitations that prevent this, perhaps nobody has just expressed interest
in this feature so far?).

From: At: Aug 27 2015 10:19:33
Subject: Re:Adding Kafka topics to a running streaming context

My streaming application needs to allow consuming new Kafka topics at arbitrary times. I know
I can stop and start the streaming context when I need to introduce a new stream, but that
seems quite disruptive. I am wondering if other people have this situation and if there is
a more elegant solution?
thanks, Yael

View raw message