lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Updated] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Thu, 28 Jun 2012 19:32:44 GMT


Michael McCandless updated LUCENE-4069:

    Attachment: LUCENE-4069-tryDeleteDocument.patch

I made various improvements to the PK lookup/update performance tester (called out as booleans
at the top) so we can test the impact of:
  * Reusing the enums
  * Pre-sorting the keys before lookup
  * Best case (flush once after each update) vs worst case
  * Using DocValues direct source instead of a stored field to hold the counter
  * Using base 36 for the PK vs base 10

Also, it's really wasteful that we go and lookup the docID by PK, to retrieve the old count,
and then do an updateDocument call which forces IW to go and to the exact same (costly) lookup

So I added a new method enabling deletion by document ID in IndexWriter.  I named it tryDeleteDocument,
and it takes SegmentInfo and int docID.  If that segment has not been merged away, the delete
succeeds, else it fails (and the app must delete through the "normal" way).  This seems to
give ~20% speedup.

I think eg Solr's "update document field by retrieving old doc, changing field, calling IW.updateDocument"
could also use this method.
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>                 Key: LUCENE-4069
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>         Attachments: BloomFilterPostingsBranch4x.patch, LUCENE-4069-tryDeleteDocument.patch,
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here:
> Benchmarks based on Wikipedia content here:
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message