lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Toke Eskildsen (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3079) Faceting module
Date Tue, 28 Jun 2011 15:37:17 GMT


Toke Eskildsen commented on LUCENE-3079:

Shai: I completely messed up the JIRA-numbers, clearly I need to go home and cool my brain.
It is fixed now. Sorry for the inconvenience.

Yes, I only created a single root (one dimension) and requested the top-5 facets. You understood
the timing measurements correctly. I am sorry that my table was confusing with regards to
memory. The first numbers was the Xmx required for index build (binary search until I got
bored), while the second if what the JVM reported after the faceting calls were finished.
For LUCENE-2369 in the middle test, the faceted search required more memory (aka higher Xmx)
than index build (which could probably have gotten by with even less).

I do not use field cance for LUCENE-2369. It holds a compressed list of ordinals for tags
for the documents in memory, with a few levels of indirections to handle doublettes. The startup
time is basically due to doublette elimination.

Regarding the memory difference, LUCENE-2369 does not operate at index-time. This means that
is it plain Lucene indexing of terms like hierarchy:a/b/c/d. Actually I am surprised that
it took 128MB for the larger test and I should probably re-run that with a lower allocation.

My guesstimage, based purely on observation, is that LUCENE-3079 requires heap relative to
the taxonomy size at indexing time. At least with the (assumedly default) settings I used.
Thus the 22M unique values in test #2 is the cause for the large memory requirement. Looking
at the number of unique tags vs. index memory requirements for case #1 and #2, the factor
seems nearly linear. It seems to fit your recommendation of splitting on large taxonomies?

I'll upload my test class for LUCENE-3079 now. I apologize for its hackish nature - this was
just meant as explorative work.

> Faceting module
> ---------------
>                 Key: LUCENE-3079
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: modules/facet
>            Reporter: Michael McCandless
>            Assignee: Shai Erera
>             Fix For: 3.4, 4.0
>         Attachments: LUCENE-3079-dev-tools.patch, LUCENE-3079.patch, LUCENE-3079.patch,
> Faceting is a hugely important feature, available in Solr today but
> not [easily] usable by Lucene-only apps.
> We should fix this, by creating a shared faceting module.
> Ideally, we factor out Solr's faceting impl, and maybe poach/merge
> from other impls (eg Bobo browse).
> Hoss describes some important challenges we'll face in doing this
> (, copied here:
> {noformat}
> To look at "faceting" as a concrete example, there are big the reasons 
> faceting works so well in Solr: Solr has total control over the 
> index, knows exactly when the index has changed to rebuild caches, has a 
> strict schema so it can make sense of field types and 
> pick faceting algos accordingly, has multi-phase distributed search 
> approach to get exact counts efficiently across multiple shards, etc...
> (and there are still a lot of additional enhancements and improvements 
> that can be made to take even more advantage of knowledge solr has because 
> it "owns" the index that we no one has had time to tackle)
> {noformat}
> This is a great list of the things we face in refactoring.  It's also
> important because, if Solr needed to be so deeply intertwined with
> caching, schema, etc., other apps that want to facet will have the
> same "needs" and so we really have to address them in creating the
> shared module.
> I think we should get a basic faceting module started, but should not
> cut Solr over at first.  We should iterate on the module, fold in
> improvements, etc., and then, once we can fully verify that cutting
> over doesn't hurt Solr (ie lose functionality or performance) we can
> later cutover.

This message is automatically generated by JIRA.
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message