lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Question about filter cache size
Date Fri, 03 Oct 2014 20:35:51 GMT
On 10/3/2014 1:57 PM, Yonik Seeley wrote:
> On Fri, Oct 3, 2014 at 3:42 PM, Peter Keegan <peterlkeegan@gmail.com> wrote:
>> Say I have a boolean field named 'hidden', and less than 1% of the
>> documents in the index have hidden=true.
>> Do both these filter queries use the same docset cache size? :
>> fq=hidden:false
>> fq=!hidden:true
> 
> Nope... !hidden:true will be smaller in the cache (it will be cached
> as hidden:true and then inverted)
> The downside is that you'll pay the cost of that inversion.

I would think that unless it's using hashDocSet, the cached data for
every filter would always be the same size.  The wiki says that
hashDocSet is no longer used for filter caching as of 1.4.0.  Is that
actually true?  Is my understanding of filterCache completely out of
touch with reality?

https://wiki.apache.org/solr/SolrCaching#The_hashDocSet_Max_Size

This does bring to mind an optimization that might help memory usage in
cases where either a very small or very large percentage of documents
match the filter: do run-length encoding on the bitset.  If the RLE
representation is at least N percent smaller than the bitset, use that
representation instead.

I think the first iteration of an RLE option would have it always on or
always off, controlled in solrconfig.xml.  A config mode where Solr
attempts RLE on every bitset and periodically reports efficiency
statistics would be pretty nice.  That data might be useful to define
default thresholds for a future automatic mode.

Thanks,
Shawn


Mime
View raw message