lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <e...@ehatchersolutions.com>
Subject Re: autowarmCount usefulness
Date Tue, 27 Jun 2006 10:44:54 GMT

On Jun 26, 2006, at 10:38 PM, Chris Hostetter wrote:
> : I'm trying to fully understand the LRUCache and the autowarmCount
> : parameter.   Why does it make sense to auto-warm filters and query
> : results?   In my case, if a new document is added it may invalidate
> : many filters, and it would require knowing the details of the
> : documents added/removed to know which caches could be copied.
> :
> : Can someone shed light on the scenarios where blindly copying over
> : any cached filters (or query results) makes sense?
>
> Autowarming of the filterCache and queryResultCache doesn't just  
> copy the
> cached values -- it reexecutes the queries used as the keys for those
> caches and generates new DocSet/DocLists using the *new* searcher,  
> before
> that searcher is made available to threads serving queries over HTTP.

Ah, that was the secret sauce I was missing.  I'm still making my way  
through the codebase understanding how it is put together, and now I  
see the regenerators in the SolrIndexSearcher for these built-in caches.

> For named User caches, autowarming doesn't work at all unless you've
> specified a regenerator -- which can do whatever it wants using the  
> new
> searcher and the information from the old cache.

How do I use LRUCache as a custom user cache to deal with cache  
misses and look up data dynamically then?   It seems to me that  
LRUCache.get() should deal with misses itself and call the  
regenerator if the key is not found.  But rather SolrIndexSearcher  
deals with this.  If I define a custom cache as an LRUCache with a  
custom regenerator, it looks like I have to add a bit more custom  
code around where I use that cache to deal with misses.  Does it make  
sense that LRUCache would pass through to a regenerator on .get() if  
the key is not found?

> The reason autowarming is configured using an autowarmCount is so  
> you can
> control just how much effort Solr should put into the autowarming  
> of the
> new cache ... if you've got a limitless supply of RAM, and an index  
> that
> doesn't change very often, you can make your caches so big that no
> DocSet/DocList is ever generated dynamically more then once -- but  
> what
> happens when your index does finally change? ... if your autowarmCount
> is the same as the size of your index, Solr could spend days  
> autowarming
> every query ever executed against your index, even if it was only  
> executed
> one time 3 weeks ago.  the autowarmCount tells Solr to only warm the N
> "best" keys in the cache where "best" is defined by the Cache
> implimentation (for an LRUCache, the "best" things are the things most
> recently used).

"LRU" abbreviation confuses me.... I see "least recently used" when I  
see that, but it really means "last recently used" within Solr.  :)

> Once upon a time Yonik and I hypothisized that it would be cool to  
> have
> autowarmTimelimit and autowarmPercentage (of current size) params  
> and some
> other things like that so you could have other ways of tweaking  
> just how
> much autowarming is done on your behalf ... but they were never built.

No worries there.  The caching is quite nice as it is.  As need  
arises, more bells and whistles can be added, but the current  
parameters are sufficient for my needs so far.

	Erik


Mime
View raw message