kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrik Kleindl <pklei...@gmail.com>
Subject Re: How to start a stream from only new records?
Date Tue, 13 Aug 2019 08:02:24 GMT
Hi

Our requirement is related, we want our streams application to only process
messages from the last x weeks.
On new deployments this requires starting the application first, stopping
the application and then resetting the offsets.
I have created https://issues.apache.org/jira/browse/KAFKA-8766 with the
idea to allow providing a custom offset selection mechanism.
No KIP has been filed yet but maybe it helps to aggregate similar cases.

best regards

Patrik

On Tue, 13 Aug 2019 at 09:26, Matthias J. Sax <matthias@confluent.io> wrote:

> You would need to delete committed offsets for the application.id and
> set `auto.offset.reset="latest" to get the behavior you want.
>
>
> -Matthias
>
> On 8/12/19 1:20 AM, Tim Ward wrote:
> > I believe not, because that only causes the application to start reading
> from latest when there is no recorded offset at application start, no?
> >
> > What I need is to be able to specify, by topic, that when the
> application starts it doesn't want to see anything other than new data,
> regardless  of what offset it committed last time it ran.
> >
> > Tim Ward
> >
> > -----Original Message-----
> > From: Boyang Chen <reluctanthero104@gmail.com>
> > Sent: 09 August 2019 17:23
> > To: users@kafka.apache.org
> > Subject: Re: How to start a stream from only new records?
> >
> > Hey Tim,
> >
> > if you are talking about avoid re-processing data and start consumption
> > from latest, you could set your `offset.reset.policy` to latest.
> >
> > Let me know if this answers your question.
> >
> > On Fri, Aug 9, 2019 at 7:09 AM Tim Ward <tim.ward@origamienergy.com>
> wrote:
> >
> >> With a real time application, nobody is interested in old data, and in
> >> particular they're not interested in paying to spend time processing it
> >> only to throw it away, thereby delaying up to date data.
> >>
> >> How do I tell StreamsBuilder.stream() to read only new data from its
> >> topic, and not process any backlog?
> >>
> >> Tim Ward
> >>
> >> This email is from Origami Energy Limited. The contents of this email
> and
> >> any attachment are confidential to the intended recipient(s). If you are
> >> not an intended recipient: (i) do not use, disclose, distribute, copy or
> >> publish this email or its contents; (ii) please contact Origami Energy
> >> Limited immediately; and then (iii) delete this email. For more
> >> information, our privacy policy is available here:
> >> https://origamienergy.com/privacy-policy/. Origami Energy Limited
> >> (company number 8619644) is a company registered in England with its
> >> registered office at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
> >>
> > This email is from Origami Energy Limited. The contents of this email
> and any attachment are confidential to the intended recipient(s). If you
> are not an intended recipient: (i) do not use, disclose, distribute, copy
> or publish this email or its contents; (ii) please contact Origami Energy
> Limited immediately; and then (iii) delete this email. For more
> information, our privacy policy is available here:
> https://origamienergy.com/privacy-policy/. Origami Energy Limited
> (company number 8619644) is a company registered in England with its
> registered office at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message