lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rob Audenaerde (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-5476) Facet sampling
Date Sun, 02 Mar 2014 14:01:22 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917414#comment-13917414
] 

Rob Audenaerde edited comment on LUCENE-5476 at 3/2/14 2:01 PM:
----------------------------------------------------------------

Thanks all for the insight!

My implementation is not correct in the fact that I indeed run over all the bits in the returning
FixedBitSet, and will work better when only looking at the set bits (the real results). The
sampling I have implemented will sample the total set of documents which is correct in the
MatchAllDocsQuery, but will behave worse and worse the fewer hits are returned; as the chance
to sample a hit decreases fast. 
 
The reason I choose to do random sampling in bins instead of just using modulo is to prevent
accidental periodicity-effects. For example, if a document is created each day and you sample
with modulo 7, you only get the 'monday' results. This can cause weird biased results. 

So my next tasks will be:

* implement {{xorshift}}, as it  seems a good improvement over the {{Math.random()}} (although
I switched to hte faster {{Random.nextInt( n )}}). 
* only sample the hits 
* remove the package 


was (Author: robau):
Thanks all for the insight!

My implementation is not correct in the fact that I indeed run over all the bits in the returning
FixedBitSet, and will work better when only looking at the set bits (the real results). The
sampling I have implemented will sample the total set of documents which is correct in the
MatchAllDocsQuery, but will behave worse and worse the fewer hits are returned; as the chance
to sample a hit decreases fast. 
 
The reason I choose to do random sampling in bins instead of just using modulo is to prevent
accidental periodicity-effects. For example, if a document is created each day and you sample
with modulo 7, you only get the 'monday' results. This can cause weird biased results. 

So my next tasks will be:

* implement {{xorshift}}, as it  seems a good improvement over the {{Math.random()}} (although
I switched to hte faster {{Random.nextInt(n)}}). 
* only sample the hits 
* remove the package 

> Facet sampling
> --------------
>
>                 Key: LUCENE-5476
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5476
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Rob Audenaerde
>         Attachments: LUCENE-5476.patch, SamplingFacetsCollector.java
>
>
> With LUCENE-5339 facet sampling disappeared. 
> When trying to display facet counts on large datasets (>10M documents) counting facets
is rather expensive, as all the hits are collected and processed. 
> Sampling greatly reduced this and thus provided a nice speedup. Could it be brought back?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message