lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-5476) Facet sampling
Date Wed, 19 Mar 2014 16:07:47 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shai Erera updated LUCENE-5476:
-------------------------------

    Attachment: LUCENE-5476.patch

I reviewed the patch more closely before I commit:

* Modified few javadocs
* Removed needsSampling() since we don't offer an extension any access to e.g. totalHits.
We can add it when there's demand.
* Fixed a bug in how carryOver was implemented -- replaced by two members {{leftoverBin}}
and {{leftoverIndex}}. So now if {{leftoverBin != -1}} we know to skip the first such documents
in the next segment and depending on {{leftoverIndex}}, whether we need to sample any of them.
Before that, we didn't really skip over the leftover docs in the bin, but started to count
a new bin.
* Added a CHANGES entry.

I reviewed the test - would be good if we can write a unit test which specifically matches
only few documents in one segment compared to the rest. I will look into it perhaps later.

I think it's ready, but if anyone wants to give createSample() another look, to make sure
this time leftover works well, I won't commit it by tomorrow anyway.

> Facet sampling
> --------------
>
>                 Key: LUCENE-5476
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5476
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Rob Audenaerde
>         Attachments: LUCENE-5476.patch, LUCENE-5476.patch, LUCENE-5476.patch, LUCENE-5476.patch,
LUCENE-5476.patch, LUCENE-5476.patch, LUCENE-5476.patch, LUCENE-5476.patch, LUCENE-5476.patch,
LUCENE-5476.patch, LUCENE-5476.patch, SamplingComparison_SamplingFacetsCollector.java, SamplingFacetsCollector.java
>
>
> With LUCENE-5339 facet sampling disappeared. 
> When trying to display facet counts on large datasets (>10M documents) counting facets
is rather expensive, as all the hits are collected and processed. 
> Sampling greatly reduced this and thus provided a nice speedup. Could it be brought back?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message