spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: pause and resume streaming app
Date Thu, 09 Jul 2015 00:26:34 GMT
Currently the only way to pause it is to stop it. The way I would do this
is use the Direct Kafka API to access the Kafka offsets, and save them to a
data store as batches finish. If you see a batch job failing because
downstream is down, stop the context. When it comes back up, start a new
streaming context using the saved offsets as starting points.


On Wed, Jul 8, 2015 at 11:52 AM, Shushant Arora <>

> Is it possible to pause and resume a streaming app?
> I have a streaming app which reads events from kafka and post to some
> external source. I want to pause the app when external source is down and
>  resume it automatically when it comes back ?
> Is it possible to pause the app and is it possible to pause only
> processing part let receiver work and brong data in memory+disk, once
> external source is up process whole data in one go.
> Thanks
> Shushant

View raw message