lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Understanding Solr memory usage
Date Mon, 02 Feb 2009 22:19:22 GMT
You shouldn't need and dont want to give tomcat anywhere near 14 of GB 
or RAM. You also should certainly not being running out of memory with 
that much RAM and that few documents. Not even close.

You want to leave plenty of RAM for the filesystem cache - so that a lot 
of that 25 gig can be cached in RAM - especially with indexes that large 
(25 gig is somewhat large by index size, 2.5 million documents is not). 
You are likely starving the filesystem cache and OS of RAM. And running 
into swap just because you have given the JVM so much RAM.

You probably do want to tune your cache sizes, but thats not your 
problem here.

Trying giving tomcat a few gig rather than 14 - the rest won't go to waste.

- Mark

Matthew A. Wagner wrote:
> I apologize in advance for what's probably a foolish question, but I'm
> trying to get a feel for how much memory a properly-configured Solr
> instance should be using.
>
> I have an index with 2.5 million documents. The documents aren't all that
> large. Our index is 25GB, and optimized fairly often.
>
> We're consistently running out of memory. Sometimes it's a heap space
> error, and other times the machine will run into swap. (The latter may not
> be directly related to Solr, but nothing else is running on the box.)
>
> We have four dedicated servers for this, each a quad Xeon with 16GB RAM. We
> have one master that receives all updates, and three slaves that handle
> queries. The three slaves have Tomcat configured for a 14GB heap. There
> really isn't a lot of disk activity.
>
> The machines seem underloaded to me, receiving less than one query per
> second on average. Requests are served in about 300ms average, so it's not
> as if we have many concurrent queries backing up.
>
> We do use multi-field faceting in some searches. I'm having a hard time
> figuring out how big of an impact this may have.
>
> None of our caches (filter, auto-warming, etc.) are set for more than 512
> documents.
>
> Obviously, memory usage is going to be very variable, but what I'm
> wondering is:
> a.) Does this sound like a sane configuration, or is something seriously
> wrong? It seems that many people are able to run considerably larger
> indexes with considerably less resources.
> b.) Is there any documentation on how the memory is being used? Is Solr
> attempting to cram as much of the 25GB index into memory as possible? Maybe
> I just overlooked something, but I don't know how to begin calculating
> Solr's memory requirements.
> c.) Does anything in the description of my Solr setup jump out at you as a
> potential source of memory problems? We've increased the heap space
> considerably, up to the current 14GB, and we're still running out of heap
> space periodically.
>
> Thanks in advance for any help!
> -- Matt Wagner
>
>   


Mime
View raw message