lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: CachingWrapperFilter: why cache per IndexReader?
Date Tue, 01 Jan 2008 20:06:06 GMT
The main reason to use a single IndexReader is because its very time 
consuming to open an IndexReader. If your index is pretty static, maybe 
this is not much of a concern. Otherwise its a major concern. But lets 
say its not...then we have to assume your going to have a huge index 
(otherwise the single thread bottleneck is just not going to matter) and 
if you have a huge index, 9 times out of 10, the bottleneck will be disk 
IO and your unlikely to benefit much from multiple Readers anyway.

Perhaps, in some esoteric case, multiple readers is the right idea 
(monster, monster, super IO system, static index?? maybe...)...but 
unless you have run into this case and have some data to show it, I 
would stick with what the community currently designates as best 
practice. Otherwise, your really just getting ahead of yourself. Single 
searcher has been scaled to some pretty huge sites. More than one 
searcher is usually responsible for very slow performance.

- Mark

Timo Nentwig wrote:
> On Tuesday 01 January 2008 19:26:51 Grant Ingersoll wrote:
>   
>> My guess would be b/c best practice is usually to only have one Reader/
>> Searcher per Directory, but I don't know if that is the real reason.
>> Most discussions/testing I have seen indicate a single Reader/Searcher
>> performs best.
>>     
>
> Well, I read this (in the FAQ) that I should use only a single 
> Searcher/Reader. And I really would like to but I ran into some FSDirectory 
> IO synchronized{} bottleneck and hence (still) don't see any alternative to 
> pooling multiple seachers (which waste a lot of memory).
>
> Even LUCENE-997 doesn't produce relief here... :-\
>
> But in general, since IndexSearcher is single-threaded I don't think that this 
> a wise approach. No matter how large the index, no matter the IO system, no 
> matter how many queries you want all this to go thru a single thread? How 
> long since does a single thread plus synchronous IO scale?
>
> However I yet didn't find any discussion on this topic so I'd be glad if 
> somebody could give me a link.
>
>   
>> -Grant
>>
>> On Jan 1, 2008, at 11:57 AM, Timo Nentwig wrote:
>>     
>>> Hi!
>>>
>>> Is there are particular reason why CachingWrapperFilter caches per
>>> IndexReader
>>> and not per IndexReader.directory()? If there are multiple
>>> IndexSearcher/IndexReader instances (and only one Directory) cache
>>> will be
>>> built and held in memory redundantly. I don't see any sense in doing
>>> so (?).
>>>
>>> Thanks for hints...
>>> Timo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>       
>> --------------------------
>> Grant Ingersoll
>> http://lucene.grantingersoll.com
>> http://www.lucenebootcamp.com
>>
>> Lucene Helpful Hints:
>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>     
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message