spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imre Nagi <>
Subject Re: Restart App and consume from checkpoint using direct kafka API
Date Thu, 31 Mar 2016 08:02:26 GMT
I'm dont know how to read the data from the checkpoint. But AFAIK and based
on my experience, I think the best thing that you can do is storing the
offset to a particular storage such as database everytime you consume the
message. Then read the offset from the database everytime you want to start
reading the message.

nb: This approach is also explained by Cody in his blog post.


On Thu, Mar 31, 2016 at 2:14 PM, vimal dinakaran <>

> Hi,
>  In the blog
> It is mentioned that enabling checkpoint works as long as the app jar is
> unchanged.
> If I want to upgrade the jar with the latest code and consume from kafka
> where it was stopped , how to do that ?
> Is there a way to read the binary object of the checkpoint during init and
> use that to start from offset ?
> Thanks
> Vimal

View raw message