kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrik Kleindl <pklei...@gmail.com>
Subject Re: Using GlobalKTable/KeyValueStore for topic cache
Date Wed, 14 Nov 2018 06:40:23 GMT
Hi Chris

We are using them like you described.
Performance is very good compared to the database used before.
Beware that until https://issues.apache.org/jira/browse/KAFKA-7380
is done the startup will be blocked until all global stores are restored (sequentially).
This can take a little for larger topic and/or multiple global stores.

We are blocking access until they are available although this is not ideal in terms of timeout
tuning.

Any ideas are welcome.

Best regards

Patrik

> Am 14.11.2018 um 00:17 schrieb Chris Toomey <ctoomey@gmail.com>:
> 
> We're considering using GlobalKTables / KeyValueStores for locally caching
> topic content in services. The topics would be compacted such that only the
> latest key/value pair would exist for a given key.
> 
> One question that's come up is how to determine, when bootstrapping the
> app, when the cache has been populated with the latest content from the
> topic (so we start with a "warm" cache). ReadOnlyKeyValueStore has
> an approximateNumEntries() method that we could use to see how much we've
> got, but trying to figure out how much there is in the topic looks much
> more difficult -- the only way I can see via the APIs / code is to use an
> AdminClient to get the topic partitions and then the KafkaConsumer to get
> the end offsets for those.
> 
> Does anyone have experience doing this kind of caching? How did you handle
> the bootstrapping issue?
> 
> Any thoughts on easier or better ways to determine when the cache is warm?
> 
> thx,
> Chris

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message