kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Markhasin <minimi...@gmail.com>
Subject Re: How to always consume from latest offset in kafka-streams
Date Sun, 21 Jan 2018 14:48:46 GMT
You can try generating a random consumer group ID to ensure that no
existing offset is used.

On 20 January 2018 at 01:11, Matthias J. Sax <matthias@confluent.io> wrote:

> That is not supported out-of-box.
>
> Configuration "auto.offset.reset" only triggers, if there are not
> committed offsets and there is KS config to change this behavior.
>
> A possible workaround might be (but not sure if I want to recommend
> this), to increase KafkaStreams commit interval via StreamsConfig (you
> could set it to MAX_VALUE, effectively disable committing). Thus,
> auto.offset.reset should trigger on restart. You might want to try it
> out and see if it works for your... Note, if we never commit, we also
> never flush KTable caches, thus you might need to disable caching, too
> (by setting cache size to zero).
>
> As an alternative, you could manipulate offsets manually before startup
> using bin/kafka-consumer-groups.sh --- the application.id is the
> group.id and and you could "seek to end" before you restart the
> application.
>
> Hope this helps.
>
>
> -Matthias
>
> On 1/19/18 9:23 AM, Saloni Vithalani wrote:
> > Our requirement is such that if a kafka-stream app is consuming a
> > partition, it should start it's consumption from latest offset of that
> > partition.
> >
> > This seems like do-able using
> >
> > streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
> "latest")
> >
> > Now, let's say using above configuration, the kafka-stream app started
> > consuming data from latest offset for a partition. And after some time,
> the
> > app crashes. When the app comes back live, we want it to consume data
> from
> > the latest offset of that partition, instead of the where it left last
> > reading.
> >
> > But I can't find anything that can help achieve it using kafka-streams
> api.
> >
> > P.S. We are using kafka-1.0.0.
> > Saloni Vithalani
> > Developer
> > Email saloniv@thoughtworks.com
> > Telephone +91 8552889571 <8552889571>
> > [image: ThoughtWorks]
> > <http://www.thoughtworks.com/?utm_campaign=saloni-vithalani-
> signature&utm_medium=email&utm_source=thoughtworks-email-
> signature-generator>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message