lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: autowarmCount usefulness
Date Tue, 27 Jun 2006 02:38:16 GMT

: I'm trying to fully understand the LRUCache and the autowarmCount
: parameter.   Why does it make sense to auto-warm filters and query
: results?   In my case, if a new document is added it may invalidate
: many filters, and it would require knowing the details of the
: documents added/removed to know which caches could be copied.
:
: Can someone shed light on the scenarios where blindly copying over
: any cached filters (or query results) makes sense?

Autowarming of the filterCache and queryResultCache doesn't just copy the
cached values -- it reexecutes the queries used as the keys for those
caches and generates new DocSet/DocLists using the *new* searcher, before
that searcher is made available to threads serving queries over HTTP.

For named User caches, autowarming doesn't work at all unless you've
specified a regenerator -- which can do whatever it wants using the new
searcher and the information from the old cache.

The documentCache doesn't support autowarming at all (because the key is
doc id, and as you say: those change with every commit).


The reason autowarming is configured using an autowarmCount is so you can
control just how much effort Solr should put into the autowarming of the
new cache ... if you've got a limitless supply of RAM, and an index that
doesn't change very often, you can make your caches so big that no
DocSet/DocList is ever generated dynamically more then once -- but what
happens when your index does finally change? ... if your autowarmCount
is the same as the size of your index, Solr could spend days autowarming
every query ever executed against your index, even if it was only executed
one time 3 weeks ago.  the autowarmCount tells Solr to only warm the N
"best" keys in the cache where "best" is defined by the Cache
implimentation (for an LRUCache, the "best" things are the things most
recently used).


Once upon a time Yonik and I hypothisized that it would be cool to have
autowarmTimelimit and autowarmPercentage (of current size) params and some
other things like that so you could have other ways of tweaking just how
much autowarming is done on your behalf ... but they were never built.



-Hoss


Mime
View raw message