lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-1278) Add optional storing of document numbers in term dictionary
Date Mon, 05 May 2008 12:33:55 GMT


Michael McCandless commented on LUCENE-1278:

It looks like the .tii file is also storing the int[] docIDs (as
inlined byte blob)?  I think that shouldn't be necessary?

This change adds a posting list like the frq file, except that it
stores only docIDs (no freq information), is stored inline in the term
dict, and includes a reader that materializes the full doc list as an
int[] instead of offering an iterator like (nextDoc()) interface

I think these changes would fit cleanly into what's been proposed for
flexible indexing.  EG, case 1a talks about storing only docID in a
posting list, here:

And recent discussions on the dev list around how to be flexible as to
which index file(s) (one or many) things are stored in, eg:

should allow you to store this data inlined into the terms dict, or as
a separate file.

Some other initial comments/questions:

  * I think this would bloat the index because the docIDs are being
    double stored (in the terms dict, and, in the frq file).  Would
    you propose changing the frq file to not store the docID when the
    term dict is doing so?

  * Why store the byte blob in the term dict, and not a separate (new)
    index file?  We lose locality for cases where one wants to iterate
    through terms and not loads these docs (eg RangeQuery).

  * Could you, instead, make a reader that reads in the full byte blob
    from the frq file for a term, and then processes that into the
    int[]?  This would require no change to indexing & the index
    format, and wouldn't waste space double-storing the docIDs.

  * I'm worried how well this scales up.  For very common terms
    allocating then decoding & holding entirely in RAM the full list
    of docIDs can become extremely costly.  Also, I don't have a clear
    sense of how apps would use the returned int[].  For example,
    would the int[] for many terms need to remain resident at the same
    time?  (Eg when running a RangeQuery).  If so, that compounds the
    scale challenge.

> Add optional storing of document numbers in term dictionary
> -----------------------------------------------------------
>                 Key: LUCENE-1278
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: lucene.1278.5.4.2008.patch, lucene.1278.5.5.2008.patch
> Add optional storing of document numbers in term dictionary.  String index field cache
and range filter creation will be faster.  
> Example read code:
> {noformat}
> TermEnum termEnum = indexReader.terms(TermEnum.LOAD_DOCS);
> do {
>   Term term = termEnum.term();
>   if (term == null || term.field() != field) break;
>   int[] docs =;
> } while (;
> {noformat}
> Example write code:
> {noformat}
> Document document = new Document();
> document.add(new Field("tag", "dog", Field.Store.YES, Field.Index.UN_TOKENIZED, Field.Term.STORE_DOCS));
> indexWriter.addDocument(document);
> {noformat}

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message