lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fuad Efendi" <f...@efendi.ca>
Subject RE: Too many open files
Date Sat, 24 Oct 2009 16:18:35 GMT

Mark, I don't understand this; of course it is use case specific, I haven't
seen any terrible behaviour with 8Gb... 32Mb is extremely small for
Nutch-SOLR -like applications, but it is acceptable for Liferay-SOLR...

Please note also, I have some documents with same IDs updated many thousands
times a day, and I believe (I hope) IndexWriter flushes "optimized" segment
instead of thousands "delete" and single "insert" in many small (32Mb) files
(especially with SOLR)...


> Hmm - came out worse than it looked. Here is a better attempt:
> 
> MergeFactor: 10
> 
> BUF   DOCS/S
> 32       37.40
> 80       39.91
> 120     40.74
> 512     38.25
> 
> Mark Miller wrote:
> > Here is an example using the Lucene benchmark package. Indexing 64,000
> > wikipedia docs (sorry for the formatting):
> >
> >      [java] ------------> Report sum by Prefix (MAddDocs) and Round (4
> > about 32 out of 256058)
> >      [java] Operation     round mrg  flush       runCnt
> > recsPerRun        rec/s  elapsedSec    avgUsedMem    avgTotalMem
> >      [java] MAddDocs_8000     0  10  32.00MB        8
> > 8000        37.40    1,711.22   124,612,472    182,689,792
> >      [java] MAddDocs_8000 -   1  10  80.00MB -  -   8 -  -  - 8000 -
> > -   39.91 -  1,603.76 - 266,716,128 -  469,925,888
> >      [java] MAddDocs_8000     2  10 120.00MB        8
> > 8000        40.74    1,571.02   348,059,488    548,233,216
> >      [java] MAddDocs_8000 -   3  10 512.00MB -  -   8 -  -  - 8000 -
> > -   38.25 -  1,673.05 - 746,087,808 -  926,089,216
> >
> > After about 32-40, you don't gain much, and it starts decreasing once
> > you start getting to high. 8GB is a terrible recommendation.
> >



Mime
View raw message