kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryanne Dolan <ryannedo...@gmail.com>
Subject Re: Using GlobalKTable/KeyValueStore for topic cache
Date Wed, 14 Nov 2018 00:54:10 GMT
Chris, consider using log compaction.

Ryanne

On Tue, Nov 13, 2018, 3:17 PM Chris Toomey <ctoomey@gmail.com> wrote:

> We're considering using GlobalKTables / KeyValueStores for locally caching
> topic content in services. The topics would be compacted such that only the
> latest key/value pair would exist for a given key.
>
> One question that's come up is how to determine, when bootstrapping the
> app, when the cache has been populated with the latest content from the
> topic (so we start with a "warm" cache). ReadOnlyKeyValueStore has
> an approximateNumEntries() method that we could use to see how much we've
> got, but trying to figure out how much there is in the topic looks much
> more difficult -- the only way I can see via the APIs / code is to use an
> AdminClient to get the topic partitions and then the KafkaConsumer to get
> the end offsets for those.
>
> Does anyone have experience doing this kind of caching? How did you handle
> the bootstrapping issue?
>
> Any thoughts on easier or better ways to determine when the cache is warm?
>
> thx,
> Chris
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message