lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Adding cores dynamically
Date Wed, 26 Aug 2009 00:24:56 GMT

: 1) We found the indexing speed starts dipping once the index grow to a
: certain size - in our case around 50G. We don't optimize, but we have
: to maintain a consistent index speed. The only way we could do that
: was keep creating new cores (on the same box, though we do use

Hmmm... it seems like ConcurrentMergeScheduler should make it possible to 
maintain semi-constant indexing speed by doing merges in background 
threads ... the only other issue would be making sure that an individual 
segment never got too big ... but that seems like it should be managable 
with the config options 

(i'm just hypothisizing, i don't normally worry about indexes of this 
size, and when i do i'm not incrementally adding to them as time goes one 
... i guess what i'm asking is if you guys ever looked into these ideas 
and dissmissed them for some reason)

: 2) Be able to drop the whole core for pruning purposes. We didn't want

that makes a lot of sense ... removing older cores is on of the only 
reaosns i could think of for this model to really make a lot of sense for 
performance reasons.

: > One problem is the IT logistics of handling the file set. At 200 million
: > records you have at least 20G of data in one Lucene index. It takes hours to
: > optimize this, and 10s of minutes to copy the optimized index around to
: > query servers.

i get that full optimizes become ridiculous at that point, but you could 
still do partial optimizes ... and isn't the total disk space with this 
strategy still the same?  Aren't you still ultimately copying the same 
amout of data arround?



-Hoss


Mime
View raw message