lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Duncan Jr <stephen.dun...@gmail.com>
Subject Re: Questions on FieldValueCache
Date Mon, 03 Aug 2009 21:45:02 GMT
On Mon, Aug 3, 2009 at 4:56 PM, Yonik Seeley <yonik@lucidimagination.com>wrote:

> On Mon, Aug 3, 2009 at 4:18 PM, Stephen Duncan
> Jr<stephen.duncan@gmail.com> wrote:
> > On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley <yonik@lucidimagination.com
> >wrote:
> > Hmm, that's a hard thing to sell to the user and my boss, as it makes the
> > query time go from nearly always being sub-second (frequently less than
> 60
> > ms), to ranging up to nearly 4 seconds for a new query not already in the
> > cache.  (My test was with 100 facets being requested, which may be
> > reasonable, as one reason to facet on a full-text field to provide a
> dynamic
> > world-cloud).
>
> Could you possibly profile it to find out what the hotspot is?
> We don't really have a good algorithm for faceting text fields, but it
> would be nice to see what the current bottleneck is.


I'll put in my TODO list to try that out soon.  I'll let you know the
results if I manage.


>
> > I guess I should update my code to use the enum method on all the fields
> > that are likely to risk crossing this line.  Should I be looking at the
> > termInstances property on the fields that are displayed in the
> > FieldValueCache on the stats page, and figuring those on the order of 10
> > million are likely to grow past the limit?
>
> For an index over 16M docs, it's perhaps closer to
> 16M/avg_bytes_per_term*256.
>
> The storage space for terms that aren't "big terms" (which come from
> the fieldCache) is 256 byte arrays, each which can be up to 16MB in
> size.  Every 65536 block of documents shares one of those byte arrays
> (or more if you have more than 16M documents).  So the average
> document can't take up more than 256 bytes in the array.  That doesn't
> mean 256 term instances though... that's the max.  The list is delta
> encoded vints, so if there are many terms, each vint could be bigger.
>
> More details in UnInvertedField after the comment:
>      //
>      // transform intermediate form into the final form, building a
> single byte[]
>      // at a time, and releasing the intermediate byte[]s as we go to avoid
>      // increasing the memory footprint.
>      //
>
> -Yonik
> http://www.lucidimagination.com
>

Ok, a lot of that is going over-my-head for the moment.  I'll try to digest
this info a little further, but for now let's see if my minimal
understanding is correct:

What will cause me to exceed the limit and fail during faceting using the fc
method is if the documents within a block of 65536 combine to take up too
much space.  And this (generally speaking) going to be a function on the
average number of unique terms in the documents?

-- 
Stephen Duncan Jr
www.stephenduncanjr.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message