lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Klaas <mike.kl...@gmail.com>
Subject Re: Help optimizing
Date Tue, 06 May 2008 02:00:19 GMT
On 3-May-08, at 10:06 AM, Daniel Andersson wrote:
>
> Our database/index is 3.5 GB and contains 4,352,471 documents. Most  
> documents are less than 1 kb. When performing a search, the results  
> vary between 1.5 seconds up to 60 seconds.
>
> I don't have a big problem with 1.5 seconds (even though below 1  
> would be nice), but 60 seconds it just.. well, scary.

That is too long, and shouldn't be happening.

>
> How do I optimize Solr to better use all the RAM? I'm using java6,  
> 64bit version, and start Solr using:
> java -Xmx7500M -Xms4096M -jar start.jar
>
> But according to top it only seems to be using 7.7% of the memory  
> (around 600 MB).

Don't try to give Solr _all_ the memory on the system.  Solr depends  
on the index existing in the OS's disk cache (this is "cached" in  
top).  You should have at least 2 GB memory for a 3.5GB index,  
depending on how much of the index is stored (best is of course to  
have 3.5GB available so it can be cached completely).

Solr will require a wide distribution of queries to "warm up" (get the  
index in the OS disk cache).   This is automatically prioritize the  
"hot spots" in the index.  If you want to load the whole thing 'cd  
datadir; cat * > /dev/null' works, but I don't recommend relying on  
that.

> Most queries are for make_id + model_id or city + state and almost  
> all of the queries are ordered by datetime_found (newest -> oldest).

How many documents match, typically?  How many documents are returned,  
typically?  How often do you commit() [I suspect frequently, based on  
the problems you are having]?

-Mike


Mime
View raw message