spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicola Ferraro <>
Subject Upgrade of Spark-Streaming application
Date Thu, 30 Jul 2015 09:07:25 GMT
I've read about the recent updates about spark-streaming integration with
Kafka (I refer to the new approach without receivers).
In the new approach, metadata are persisted in checkpoint folders on HDFS
so that the SparkStreaming context can be recreated in case of failures.
This means that the streaming application will restart from the where it
exited and the message consuming process continues with new messages only.
Also, if I manually stop the streaming process and recreate the context
from checkpoint (using an approach similar to,
the behavior would be the same.

Now, suppose I want to change something in the software and modify the
processing pipeline.
Can spark use the previous checkpoint to recreate the new application? Will
I ever be able to upgrade the software without processing all the messages
in Kafka again?


View raw message