lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaeger, Jay - DOT" <Jay.Jae...@dot.wi.gov>
Subject RE: SOLR Index Speed
Date Mon, 26 Sep 2011 18:57:59 GMT
500 / second would be 1,800,000 per hour (much more than 500K documents).

1)  how big is each document?
2)  how big are your index files?
3)  as others have recently written, make sure you don't give your JRE so much memory that
your OS is starved for memory to use for file system cache.

JRJ

-----Original Message-----
From: Lord Khan Han [mailto:khanuniverse1@gmail.com] 
Sent: Monday, September 26, 2011 6:09 AM
To: solr-user@lucene.apache.org
Subject: SOLR Index Speed

Hi,

We have 500K web document and usind solr (trunk) to index it. We have
special anaylizer which little bit heavy cpu .
Our machine config:

32 x cpu
32 gig ram
SAS HD

We are sending document with 16 reduce client (from hadoop) to the stand
alone solr server. the problem is we couldnt get speedier than the 500 doc /
per sec. 500K document tooks 7-8 hours to index :(

While indexin the the solr server cpu load is around : 5-6  (32 max)  it
means  %20 of the cpu total power. We have plenty ram ...

I turned of auto commit  and give 8198 rambuffer .. there is no io wait ..

How can I make it faster ?

PS: solr streamindex  is not option because we need to submit javabin...

thanks..

Mime
View raw message