kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Bejeck <b...@confluent.io>
Subject Re: Using GlobalKTable/KeyValueStore for topic cache
Date Wed, 14 Nov 2018 03:50:50 GMT
Hi Chris

Yes, for every state store in a kafka streams application, the state
restore listener is executed.

-Bill

On Tue, Nov 13, 2018 at 10:36 PM Chris Toomey <ctoomey@gmail.com> wrote:

> Thanks Bill. So poking around the code a bit, it looks like perhaps any
> kafka streams execution that produces a state store would execute this
> "restore state store" operation, even creating a new state store, is that
> correct? If so that indeed could be just what I need.
>
>
> On Tue, Nov 13, 2018 at 6:38 PM Bill Bejeck <bill@confluent.io> wrote:
>
> > Hi Chris,
> >
> > I'm not sure I totally understand your requirements but the
> > StateRestoreListener (
> >
> >
> https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/processor/StateRestoreListener.html
> > )
> > class  provides callbacks when restoring state stores (including
> > GlobalStores) and may provide what you are looking for.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Nov 13, 2018 at 8:49 PM Chris Toomey <ctoomey@gmail.com> wrote:
> >
> > > Definitely Ryanne -- that's what I meant by "topics would be
> compacted".
> > >
> > > But that doesn't obviate checking bootstrapping progress.
> > >
> > > On Tue, Nov 13, 2018 at 5:04 PM Ryanne Dolan <ryannedolan@gmail.com>
> > > wrote:
> > >
> > > > Chris, consider using log compaction.
> > > >
> > > > Ryanne
> > > >
> > > > On Tue, Nov 13, 2018, 3:17 PM Chris Toomey <ctoomey@gmail.com>
> wrote:
> > > >
> > > > > We're considering using GlobalKTables / KeyValueStores for locally
> > > > caching
> > > > > topic content in services. The topics would be compacted such that
> > only
> > > > the
> > > > > latest key/value pair would exist for a given key.
> > > > >
> > > > > One question that's come up is how to determine, when bootstrapping
> > the
> > > > > app, when the cache has been populated with the latest content from
> > the
> > > > > topic (so we start with a "warm" cache). ReadOnlyKeyValueStore has
> > > > > an approximateNumEntries() method that we could use to see how much
> > > we've
> > > > > got, but trying to figure out how much there is in the topic looks
> > much
> > > > > more difficult -- the only way I can see via the APIs / code is to
> > use
> > > an
> > > > > AdminClient to get the topic partitions and then the KafkaConsumer
> to
> > > get
> > > > > the end offsets for those.
> > > > >
> > > > > Does anyone have experience doing this kind of caching? How did you
> > > > handle
> > > > > the bootstrapping issue?
> > > > >
> > > > > Any thoughts on easier or better ways to determine when the cache
> is
> > > > warm?
> > > > >
> > > > > thx,
> > > > > Chris
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message