spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudarshan Kadambi (BLOOMBERG/ 731 LEX)" <>
Subject Re: Adding Kafka topics to a running streaming context
Date Thu, 27 Aug 2015 17:48:55 GMT
I put up Let's continue this conversation

From: At: Aug 27 2015 10:30:12
To: Sudarshan Kadambi (BLOOMBERG/ 731 LEX)
Subject: Re: Adding Kafka topics to a running streaming context

If you all can say a little more about what your requirements are, maybe we can get a jira

I think the easiest way to deal with this currently is to start a new job before stopping
the old one, which should prevent latency problems.

On Thu, Aug 27, 2015 at 9:24 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEX) <>

This is something we have been needing for a while too. We are restarting the streaming context
to handle new topic subscriptions & unsubscriptions which affects latency of update handling.
I think this is something that needs to be addressed in core Spark Streaming (I can't think
of any fundamental limitations that prevent this, perhaps nobody has just expressed interest
in this feature so far?).

From: At: Aug 27 2015 10:19:33
Subject: Re:Adding Kafka topics to a running streaming context

My streaming application needs to allow consuming new Kafka topics at arbitrary times. I know
I can stop and start the streaming context when I need to introduce a new stream, but that
seems quite disruptive. I am wondering if other people have this situation and if there is
a more elegant solution?
thanks, Yael

View raw message