kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Toomey <ctoo...@gmail.com>
Subject Re: Using GlobalKTable/KeyValueStore for topic cache
Date Wed, 14 Nov 2018 01:49:20 GMT
Definitely Ryanne -- that's what I meant by "topics would be compacted".

But that doesn't obviate checking bootstrapping progress.

On Tue, Nov 13, 2018 at 5:04 PM Ryanne Dolan <ryannedolan@gmail.com> wrote:

> Chris, consider using log compaction.
>
> Ryanne
>
> On Tue, Nov 13, 2018, 3:17 PM Chris Toomey <ctoomey@gmail.com> wrote:
>
> > We're considering using GlobalKTables / KeyValueStores for locally
> caching
> > topic content in services. The topics would be compacted such that only
> the
> > latest key/value pair would exist for a given key.
> >
> > One question that's come up is how to determine, when bootstrapping the
> > app, when the cache has been populated with the latest content from the
> > topic (so we start with a "warm" cache). ReadOnlyKeyValueStore has
> > an approximateNumEntries() method that we could use to see how much we've
> > got, but trying to figure out how much there is in the topic looks much
> > more difficult -- the only way I can see via the APIs / code is to use an
> > AdminClient to get the topic partitions and then the KafkaConsumer to get
> > the end offsets for those.
> >
> > Does anyone have experience doing this kind of caching? How did you
> handle
> > the bootstrapping issue?
> >
> > Any thoughts on easier or better ways to determine when the cache is
> warm?
> >
> > thx,
> > Chris
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message