kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ward <tim.w...@origamienergy.com.INVALID>
Subject RE: How to start a stream from only new records?
Date Mon, 12 Aug 2019 08:20:58 GMT
I believe not, because that only causes the application to start reading from latest when there
is no recorded offset at application start, no?

What I need is to be able to specify, by topic, that when the application starts it doesn't
want to see anything other than new data, regardless  of what offset it committed last time
it ran.

Tim Ward

-----Original Message-----
From: Boyang Chen <reluctanthero104@gmail.com>
Sent: 09 August 2019 17:23
To: users@kafka.apache.org
Subject: Re: How to start a stream from only new records?

Hey Tim,

if you are talking about avoid re-processing data and start consumption
from latest, you could set your `offset.reset.policy` to latest.

Let me know if this answers your question.

On Fri, Aug 9, 2019 at 7:09 AM Tim Ward <tim.ward@origamienergy.com> wrote:

> With a real time application, nobody is interested in old data, and in
> particular they're not interested in paying to spend time processing it
> only to throw it away, thereby delaying up to date data.
>
> How do I tell StreamsBuilder.stream() to read only new data from its
> topic, and not process any backlog?
>
> Tim Ward
>
> This email is from Origami Energy Limited. The contents of this email and
> any attachment are confidential to the intended recipient(s). If you are
> not an intended recipient: (i) do not use, disclose, distribute, copy or
> publish this email or its contents; (ii) please contact Origami Energy
> Limited immediately; and then (iii) delete this email. For more
> information, our privacy policy is available here:
> https://origamienergy.com/privacy-policy/. Origami Energy Limited
> (company number 8619644) is a company registered in England with its
> registered office at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
>
This email is from Origami Energy Limited. The contents of this email and any attachment are
confidential to the intended recipient(s). If you are not an intended recipient: (i) do not
use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
Mime
View raw message