lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Britske <gbr...@gmail.com>
Subject Re: big discrepancy between elapsedtime and qtime although enableLazyFieldLoading= true
Date Mon, 28 Jul 2008 21:32:37 GMT

I'm using the solr-nightly of 2008-04-05



Grant Ingersoll-6 wrote:
> 
> What version of Solr/Lucene are you using?
> 
> On Jul 28, 2008, at 4:53 PM, Britske wrote:
> 
>>
>> I'm on a development box currently and production servers will be  
>> bigger, but
>> at the same time the index will be too.
>>
>> Each query requests at most 20 stored fields. Why doesn't help
>> lazyfieldloading in this situation?
>> I don't need to retrieve all stored fields and I thought I wasn't  
>> doing this
>> (through limiting the fields returned using the FL-param), but if I  
>> read
>> your comment correctly, apparently I am retrieving them all, I'm  
>> just not
>> displaying them all?
>>
>> Also, if I understand correctly, for optimal performance I need to  
>> have at
>> least enough RAM to put the entire Index size in OS cache (thus RAM)  
>> + the
>> amount of RAM that SOLR / Lucene consumes directly through the JVM?  
>> (which
>> among other things includes the Lucene field-cache + all of SOlr's  
>> caches on
>> top of that).
>>
>> I've never read the requirement of having the entire index in OS cache
>> before, is this because in normal situations (with less stored  
>> fields) it
>> doesn't matter much? I'm just surprised to hear of this for the  
>> first time,
>> since it will likely give a big impact on my design.
>>
>> Luckily most of the normal queries return 10 documents each, which  
>> results
>> in a discrepancy between total elapsed time and qTIme of about 15-30  
>> ms.
>> Doesn't this seem strange, since to me it would seem logical that the
>> discrepancy would be at least 1/10th of fetching 100 documents.
>>
>> hmm, hope you can shine some light on this,
>>
>> Thanks a lot,
>> Britske
>>
>>
>>
>> Yonik Seeley wrote:
>>>
>>> That's a bit too tight to have *all* of the index cached...your best
>>> bet is to go to 4GB+, or figure out a way not to have to retrieve so
>>> many stored fields.
>>>
>>> -Yonik
>>>
>>> On Mon, Jul 28, 2008 at 4:27 PM, Britske <gbrits@gmail.com> wrote:
>>>>
>>>> Size on disk is 1.84 GB (of which 1.3 GB sits in FDT files if that
>>>> matters)
>>>> Physical RAM is 2 GB with -Xmx800M set to Solr.
>>>>
>>>>
>>>> Yonik Seeley wrote:
>>>>>
>>>>> That high of a difference is due to the part of the index  
>>>>> containing
>>>>> these particular stored fields not being in OS cache.  What's the  
>>>>> size
>>>>> on disk of your index compared to your physical RAM?
>>>>>
>>>>> -Yonik
>>>>>
>>>>> On Mon, Jul 28, 2008 at 4:10 PM, Britske <gbrits@gmail.com> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> For some queries I need to return a lot of rows at once (say 100).
>>>>>> When performing these queries I notice a big difference between 

>>>>>> qTime
>>>>>> (which
>>>>>> is mostly in the 15-30 ms range due to caching) and total time  
>>>>>> taken to
>>>>>> return the response (measured through SolrJ's elapsedTime),  
>>>>>> which takes
>>>>>> between 500-1600 ms.
>>>>>>
>>>>>> For queries which return less rows the difference becomes less  
>>>>>> big.
>>>>>>
>>>>>> I presume (after reading some threads in the past) that this is 

>>>>>> due to
>>>>>> solr
>>>>>> constructing and streaming the response (which includes  
>>>>>> retrieving the
>>>>>> stored fields) , which is something that is not calculated in  
>>>>>> qTime.
>>>>>>
>>>>>> Documents have a lot of stored fields (more than 10.000), but at
 
>>>>>> any
>>>>>> given
>>>>>> query a maximum of say 20 are returned (through fl-field ) or  
>>>>>> used (as
>>>>>> part
>>>>>> of filtering, faceting, sorting)
>>>>>>
>>>>>> I would have thought that enabling enableLazyFieldLoading for this
>>>>>> situation
>>>>>> would mean a lot, since so many stored fields can be skipped,  
>>>>>> but I
>>>>>> notice
>>>>>> no real difference in measuring total elapsed time (or qTime for
 
>>>>>> that
>>>>>> matter).
>>>>>>
>>>>>> Am I missing something here? What criteria would need to be met 

>>>>>> for a
>>>>>> field
>>>>>> to not be loaded for instance? Should I see a big performance  
>>>>>> boost in
>>>>>> this
>>>>>> situation?
>>>>>>
>>>>>> Thanks,
>>>>>> Britske
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698590.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698909.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699550.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
> 
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
> 
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
> 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699992.html
Sent from the Solr - User mailing list archive at Nabble.com.


Mime
View raw message