lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <yo...@apache.org>
Subject Re: big discrepancy between elapsedtime and qtime although enableLazyFieldLoading= true
Date Mon, 28 Jul 2008 21:06:17 GMT
On Mon, Jul 28, 2008 at 4:53 PM, Britske <gbrits@gmail.com> wrote:
> Each query requests at most 20 stored fields. Why doesn't help
> lazyfieldloading in this situation?

It's the disk seek that kills you... loading 1 byte or 1000 bytes per
document would be about the same speed.

> Also, if I understand correctly, for optimal performance I need to have at
> least enough RAM to put the entire Index size in OS cache (thus RAM) + the
> amount of RAM that SOLR / Lucene consumes directly through the JVM?

The normal usage is to just retrieve the stored fields for the top 10
(or a window of 10 or 20) documents.  Under this scenario, the
slowdown from not having all of the stored fields cached is usually
acceptable.  Faster disks (seek time) can also help.

> Luckily most of the normal queries return 10 documents each, which results
> in a discrepancy between total elapsed time and qTIme of about 15-30 ms.
> Doesn't this seem strange, since to me it would seem logical that the
> discrepancy would be at least 1/10th of fetching 100 documents.

Yes, in general 1/10th the cost is what one would expect on average.
But some of the docs you are trying to retrieve *will* be in cache, so
it's hard to control this test.
You could try forcing the index out of memory by "cat"ing some other
big files multiple times and then re-trying.... or do a reboot to be
sure.

-Yonik

Mime
View raw message