lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: big discrepancy between elapsedtime and qtime although enableLazyFieldLoading= true
Date Mon, 28 Jul 2008 21:06:51 GMT
What version of Solr/Lucene are you using?

On Jul 28, 2008, at 4:53 PM, Britske wrote:

>
> I'm on a development box currently and production servers will be  
> bigger, but
> at the same time the index will be too.
>
> Each query requests at most 20 stored fields. Why doesn't help
> lazyfieldloading in this situation?
> I don't need to retrieve all stored fields and I thought I wasn't  
> doing this
> (through limiting the fields returned using the FL-param), but if I  
> read
> your comment correctly, apparently I am retrieving them all, I'm  
> just not
> displaying them all?
>
> Also, if I understand correctly, for optimal performance I need to  
> have at
> least enough RAM to put the entire Index size in OS cache (thus RAM)  
> + the
> amount of RAM that SOLR / Lucene consumes directly through the JVM?  
> (which
> among other things includes the Lucene field-cache + all of SOlr's  
> caches on
> top of that).
>
> I've never read the requirement of having the entire index in OS cache
> before, is this because in normal situations (with less stored  
> fields) it
> doesn't matter much? I'm just surprised to hear of this for the  
> first time,
> since it will likely give a big impact on my design.
>
> Luckily most of the normal queries return 10 documents each, which  
> results
> in a discrepancy between total elapsed time and qTIme of about 15-30  
> ms.
> Doesn't this seem strange, since to me it would seem logical that the
> discrepancy would be at least 1/10th of fetching 100 documents.
>
> hmm, hope you can shine some light on this,
>
> Thanks a lot,
> Britske
>
>
>
> Yonik Seeley wrote:
>>
>> That's a bit too tight to have *all* of the index cached...your best
>> bet is to go to 4GB+, or figure out a way not to have to retrieve so
>> many stored fields.
>>
>> -Yonik
>>
>> On Mon, Jul 28, 2008 at 4:27 PM, Britske <gbrits@gmail.com> wrote:
>>>
>>> Size on disk is 1.84 GB (of which 1.3 GB sits in FDT files if that
>>> matters)
>>> Physical RAM is 2 GB with -Xmx800M set to Solr.
>>>
>>>
>>> Yonik Seeley wrote:
>>>>
>>>> That high of a difference is due to the part of the index  
>>>> containing
>>>> these particular stored fields not being in OS cache.  What's the  
>>>> size
>>>> on disk of your index compared to your physical RAM?
>>>>
>>>> -Yonik
>>>>
>>>> On Mon, Jul 28, 2008 at 4:10 PM, Britske <gbrits@gmail.com> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> For some queries I need to return a lot of rows at once (say 100).
>>>>> When performing these queries I notice a big difference between  
>>>>> qTime
>>>>> (which
>>>>> is mostly in the 15-30 ms range due to caching) and total time  
>>>>> taken to
>>>>> return the response (measured through SolrJ's elapsedTime),  
>>>>> which takes
>>>>> between 500-1600 ms.
>>>>>
>>>>> For queries which return less rows the difference becomes less  
>>>>> big.
>>>>>
>>>>> I presume (after reading some threads in the past) that this is  
>>>>> due to
>>>>> solr
>>>>> constructing and streaming the response (which includes  
>>>>> retrieving the
>>>>> stored fields) , which is something that is not calculated in  
>>>>> qTime.
>>>>>
>>>>> Documents have a lot of stored fields (more than 10.000), but at  
>>>>> any
>>>>> given
>>>>> query a maximum of say 20 are returned (through fl-field ) or  
>>>>> used (as
>>>>> part
>>>>> of filtering, faceting, sorting)
>>>>>
>>>>> I would have thought that enabling enableLazyFieldLoading for this
>>>>> situation
>>>>> would mean a lot, since so many stored fields can be skipped,  
>>>>> but I
>>>>> notice
>>>>> no real difference in measuring total elapsed time (or qTime for  
>>>>> that
>>>>> matter).
>>>>>
>>>>> Am I missing something here? What criteria would need to be met  
>>>>> for a
>>>>> field
>>>>> to not be loaded for instance? Should I see a big performance  
>>>>> boost in
>>>>> this
>>>>> situation?
>>>>>
>>>>> Thanks,
>>>>> Britske
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698590.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698909.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699550.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ








Mime
View raw message