kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: How to start a stream from only new records?
Date Tue, 13 Aug 2019 07:25:03 GMT
You would need to delete committed offsets for the application.id and
set `auto.offset.reset="latest" to get the behavior you want.


-Matthias

On 8/12/19 1:20 AM, Tim Ward wrote:
> I believe not, because that only causes the application to start reading from latest
when there is no recorded offset at application start, no?
> 
> What I need is to be able to specify, by topic, that when the application starts it doesn't
want to see anything other than new data, regardless  of what offset it committed last time
it ran.
> 
> Tim Ward
> 
> -----Original Message-----
> From: Boyang Chen <reluctanthero104@gmail.com>
> Sent: 09 August 2019 17:23
> To: users@kafka.apache.org
> Subject: Re: How to start a stream from only new records?
> 
> Hey Tim,
> 
> if you are talking about avoid re-processing data and start consumption
> from latest, you could set your `offset.reset.policy` to latest.
> 
> Let me know if this answers your question.
> 
> On Fri, Aug 9, 2019 at 7:09 AM Tim Ward <tim.ward@origamienergy.com> wrote:
> 
>> With a real time application, nobody is interested in old data, and in
>> particular they're not interested in paying to spend time processing it
>> only to throw it away, thereby delaying up to date data.
>>
>> How do I tell StreamsBuilder.stream() to read only new data from its
>> topic, and not process any backlog?
>>
>> Tim Ward
>>
>> This email is from Origami Energy Limited. The contents of this email and
>> any attachment are confidential to the intended recipient(s). If you are
>> not an intended recipient: (i) do not use, disclose, distribute, copy or
>> publish this email or its contents; (ii) please contact Origami Energy
>> Limited immediately; and then (iii) delete this email. For more
>> information, our privacy policy is available here:
>> https://origamienergy.com/privacy-policy/. Origami Energy Limited
>> (company number 8619644) is a company registered in England with its
>> registered office at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
>>
> This email is from Origami Energy Limited. The contents of this email and any attachment
are confidential to the intended recipient(s). If you are not an intended recipient: (i) do
not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
> 


Mime
View raw message