kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Toomey <ctoo...@gmail.com>
Subject Using GlobalKTable/KeyValueStore for topic cache
Date Tue, 13 Nov 2018 23:17:34 GMT
We're considering using GlobalKTables / KeyValueStores for locally caching
topic content in services. The topics would be compacted such that only the
latest key/value pair would exist for a given key.

One question that's come up is how to determine, when bootstrapping the
app, when the cache has been populated with the latest content from the
topic (so we start with a "warm" cache). ReadOnlyKeyValueStore has
an approximateNumEntries() method that we could use to see how much we've
got, but trying to figure out how much there is in the topic looks much
more difficult -- the only way I can see via the APIs / code is to use an
AdminClient to get the topic partitions and then the KafkaConsumer to get
the end offsets for those.

Does anyone have experience doing this kind of caching? How did you handle
the bootstrapping issue?

Any thoughts on easier or better ways to determine when the cache is warm?

thx,
Chris

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message