lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanne Grinovero <>
Subject Re: Document retrieval, performance, and DocValues
Date Tue, 05 Jul 2016 16:38:31 GMT
Hi Randy,

a first quick and easy win would be to rewrite it as:

DocumentStoredFieldVisitor visitor = new
for(int i=0; i<docs.scoreDocs.length; i++) {
   ScoreDoc sdoc = docs.scoreDocs[i];
   String id = indexReader.document(sdoc.doc,
    // … retrieve data with id

as creating all those FieldVisitor instances isn't free.

Next, I'd suggest to try optimise this loop further without measuring
the "retrieve data with id" aspect?
It's maybe possible to just fill in an array of those ids, and load
them all as a second step, in batches?

I'm not sure about your other questions, will leave that to others on the list.

-- Sanne

On 5 July 2016 at 16:20, Randy Tidd <> wrote:
> My Lucene index has about 3 million documents and result sets can be large, often 1000’s
and sometimes as many as 100,000.  I am expecting the index size to grow 5-10x as the system
> I index 5 fields, and per recommendations I’ve read, am storing the minimal data in
Lucene, currently just a 12 byte numeric identifier (a Mongo ObjectId) per document.  I store
the rest of the data separately and use the id I get from Lucene to look it up there.
> In my load testing, a search like this:
>     TopDocs docs =, maxResults, sort)
> takes about 50-75 msec which is good.  Retrieving documents with a loop like this:
>     for(int i=0; i<docs.scoreDocs.length; i++) {
>         ScoreDoc sdoc = docs.scoreDocs[i];
>         String id = indexReader.document(sdoc.doc, Collections.singleton("pos_id”)).getField("pos_id").stringValue();
>         // … retrieve data with id
>     }
> takes around 350-400 msec, sometimes as long as 800 msec.  I’m looking for ways to
try to decrease this time if possible.
> I’ve read up on DocValues and am not sure if that is intended to help with this.  I
understand that it is a separate store/mapping of Lucene’s internal document id’s to my
“pos_id” which sounds like it may help but I am not sure.  I tried getting the id’s
from my reader like this:
>             String id = MultiDocValues.getBinaryValues(indexReader, "pos_id").get(sdoc.doc).utf8ToString()
> But performance was no better.  However I saw in the docs for MultiDocValues that I may
get better performance using the "atomic leaves and then operate per-LeafReader”. I searched
around and could not find documentation on how to do that. I see some examples using leaf
readers in the solr projects but they were just examples and I don’t think were written
specifically to optimize performance.  It would be great to find an explanation of why there
are multiple leaf readers per reader and how to use them.
> So my questions are 1) are DocValues a possibility for improving my document retrieval
performance, and 2) if so, where can I find an example of this that is written for best performance?
> Thanks in advance!
> Randy
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message