lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: How much disk space does optimize really take
Date Wed, 07 Oct 2009 17:40:37 GMT
I think that argument requires auto commit to be on and opening readers
after the optimize starts? Otherwise, the optimized version is not put
into place until a commit is called, and a Reader won't see the newly
merged segments until then - so the original index is kept around in
either case - having a Reader open on it shouldn't affect the space
requirements?

Yonik Seeley wrote:
> On Wed, Oct 7, 2009 at 12:51 PM, Phillip Farber <pfarber@umich.edu> wrote:
>   
>> In a separate thread, I've detailed how an optimize is taking > 2x disk
>> space. We don't use solr distribution/snapshooter.  We are using the default
>> deletion policy = 1. We can't optimize a 192G index in 400GB of space.
>>
>> This thread in lucene/java-user
>>
>> http://www.gossamer-threads.com/lists/lucene/java-user/43475
>>
>> suggests that an optimize should not take > 2x unless perhaps an IndexReader
>> is holding on to segments. This could be our problem since when optimization
>> runs out of space, if we stop tomcat, a number of files go away and space is
>> recovered.
>>
>> But we are not searching the index so how could a Searcher/IndexReader have
>> any segments open?
>>
>> I notice in the logs that as part of routine commits or as part of optimize
>> a Searcher is registered and autowarmed from a previous searcher (of course
>> there's nothing in the caches -- this is just a build machine).
>>
>> INFO: registering core:
>> Oct 6, 2009 2:16:20 PM org.apache.solr.core.SolrCore registerSearcher
>> INFO: [] Registered new searcher Searcher@2e097617 main
>>
>> Does this means that there's always a lucene IndexReader holding segment
>> files open so they can't be deleted during an optimize so we run out of disk
>> space > 2x?
>>     
>
> Yes.
> A feature could probably now be developed now that avoids opening a
> reader until it's requested.
> That wasn't really possible in the past - due to many issues such as
> Lucene autocommit.
>
> -Yonik
> http://www.lucidimagination.com
>   


-- 
- Mark

http://www.lucidimagination.com




Mime
View raw message