lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <wun...@wunderwood.org>
Subject Re: index size before and after commit
Date Thu, 01 Oct 2009 14:36:42 GMT
I've now worked on three different search engines and they all have a  
3X worst
case on space, so I'm familiar with this case. --wunder

On Oct 1, 2009, at 7:15 AM, Mark Miller wrote:

> Nice one ;) Its not technically a case where optimize requires > 2x
> though in case the user asking gets confused. Its a case unrelated to
> optimize that can grow your index. Then you need < 2x for the  
> optimize,
> since you won't copy the deletes.
>
> It also requires that you jump hoops to delete everything. If you  
> delete
> everything with *:*, that is smart enough not to just do a delete on
> every document - it just creates a new index, allowing the removal of
> the old very efficiently.
>
> Def agree on the more disk space.
>
> Walter Underwood wrote:
>> Here is how you need 3X. First, index everything and optimize. Then
>> delete everything and reindex without any merges.
>>
>> You have one full-size index containing only deleted docs, one
>> full-size index containing reindexed docs, and need that much space
>> for a third index.
>>
>> Honestly, disk is cheap, and there is no way to make Lucene work
>> reliably with less disk. 1TB is a few hundred dollars. You have a  
>> free
>> search engine, buy some disk.
>>
>> wunder
>>
>> On Oct 1, 2009, at 6:25 AM, Grant Ingersoll wrote:
>>
>>>> 151GB or as little as from 183GB to 182GB.  Is that size after a
>>>> commit close to the size the index would be after an optimize?  For
>>>> that matter, are there cases where optimization can take more than
>>>> 2x?  I've heard of cases but have not observed them in my system.
>>>
>>> I seem to recall a case where it can be 3x, but I don't know that it
>>> has been observed much.
>>
>
>
> -- 
> - Mark
>
> http://www.lucidimagination.com
>
>
>


Mime
View raw message