spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: Upgrades of streaming jobs
Date Fri, 09 Mar 2018 22:24:09 GMT
Yes, all checkpoints are forward compatible.

However, you do need to restart the query if you want to update the code of
the query. This downtime can be in less than a second (if you just restart
the query without stopping the application/Spark driver) or 10s of seconds
(if you have to stop the application and resubmit your application to the

On Thu, Mar 8, 2018 at 12:11 PM, Georg Heiler <>

> Hi
> What is the state of spark structured streaming jobs and upgrades?
> Can checkpoints of version 1 be read by version 2 of a job? Is downtime
> required to upgrade the job?
> Thanks

View raw message