lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: useCompoundFile, mergeFactor & index replication
Date Tue, 10 Jul 2007 00:49:46 GMT
Using the compound index format will, I imagine, result in fewer and larger files being sent
over the wire during replication.
I would stay with the multi-file index format and increase the max number of open files via
ulimit or sysctl (Linux, at least).

Otis
--
Lucene Consulting -- http://lucene-consulting.com/



----- Original Message ----
From: James Jory <james@vinome.com>
To: solr-user@lucene.apache.org
Sent: Tuesday, July 10, 2007 1:06:26 AM
Subject: useCompoundFile, mergeFactor & index replication

I am working with a pre-production Solr installation using a single 
master and 4 search slaves. The index will eventually contain ~20M 
documents. There are about 25 fields in the index but only a few 
"integer" and small "string" fields actually have their data stored in 
the index. The index is being updated constantly so snapshot replication 
will be frequent. The plan is to optimize once per day but that can be 
changed.

During our testing we are running into the "too many open files" 
exception. We are going to change the ulimit on all the servers but I 
was curious if anyone is enabling the compound file format and/or 
lowering the merge factor to keep the file count down and, if so, what 
impact those changes are having on replication. So far in our testing we 
haven't noticed any significant differences in replication 
performance/behavior but we haven't grown the index much yet.

Does anyone have any experience or advice on using the compound file 
format with index replication?

Thanks,
James





Mime
View raw message