lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Fairey" <sifai...@gmail.com>
Subject RE: Solr configuration, memory usage and MMapDirectory
Date Wed, 08 Oct 2014 20:42:24 GMT
Hi

Thanks for this I will investigate further after reading a number of your points in more detail,
I do have a feeling they've setup too many entries in the filter cache (1000s) so will revisit
that.

Just a note on numbers, those were valid when I made the post but obviously they change as
the week progresses before a regular clean-up of content, current numbers for info (if it's
at all relevant) from the index admin view on one of the 2 nodes is:

Last Modified:	18 minutes ago
Num Docs:    	24590368
Max Doc:    	29139255
Deleted Docs: 	4548887
Version:  		1297982
Segment Count: 	28
	
 	           Version 	    Gen 	Size
Master: 	1412798583558 402364 52.98 GB

Top:
2996 tomcat6   20   0  189g  73g 1.5g S   15 58.7  58034:04 java

And the only GC option I can see that is on is "- XX:+UseConcMarkSweepGC"

Regarding the XY problem, you are very likely correct, unfortunately I wasn't involved in
the config and I very much suspect when it was done many of the defaults were used and then
if it didn't work or there was say an out of memory error they just upped the heap to solve
the symptom without investigating the cause. The luxury of having more than enough RAM I guess!

I'm going to get some late night downtime soon at which point I'm hoping to change the heap
size, GC settings and add the JMX, it's not exposed to the internet so no security is fine.

Right off to do some reading!

Cheers

Si

-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org] 
Sent: 08 October 2014 21:09
To: solr-user@lucene.apache.org
Subject: Re: Solr configuration, memory usage and MMapDirectory

On 10/8/2014 4:02 AM, Simon Fairey wrote:
> I'm currently setting up jconsole but as I have to remotely monitor (no gui capability
on the server) I have to wait before I can restart solr with a JMX port setup. In the meantime
I looked at top and given the calculations you said based on your top output and this top
of my java process from the node that handles the querying, the indexing node has a similar
memory profile:
> 
> https://www.dropbox.com/s/pz85dm4e7qpepco/SolrTop.png?dl=0
> 
> It would seem I need a monstrously large heap in the 60GB region?
> 
> We do use a lot of navigators/filters so I have set the caches to be quite large for
these, are these what are using up the memory?

With a VIRT size of 189GB and a RES size of 73GB, I believe you probably have more than 45GB
of index data.  This might be a combination of old indexes and the active index.  Only the
indexes (cores) that are being actively used need to be considered when trying to calculate
the total RAM needed.  Other indexes will not affect performance, even though they increase
your virtual memory size.

With MMap, part of the virtual memory size is the size of the index data that has been opened
on the disk.  This is not memory that's actually allocated.  There's a very good reason that
mmap has been the default in Lucene and Solr for more than two years.

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

You stated originally that you have 25 million document and 45GB of index data on each node.
 With those numbers and a conservative configuration, I would expect that you need about 4GB
of heap, maybe as much as 8GB.  I cannot think of any reason that you would NEED a heap 60GB
or larger.

Each field that you sort on, each field that you facet on with the default facet.method of
fc, and each filter that you cache will use a large block of memory.  The size of that block
of memory is almost exclusively determined by the number of documents in the index.

With 25 million documents, each filterCache entry will be approximately 3MB -- one bit for
every document.  I do not know how big each FieldCache entry is for a sort field and a facet
field, but assume that they are probably larger than the 3MB entries on the filterCache.

I've got a filterCache sized at 64, with an autowarmCount of 4.  With larger autowarmCount
values, I was seeing commits take 30 seconds or more, because each of those filters can take
a few seconds to execute.
Cache sizes in the thousands are rarely necessary, and just chew up a lot of memory with no
benefit.  Large autowarmCount values are also rarely necessary.  Every time a new searcher
is opened by a commit, add up all your autowarmCount values and realize that the searcher
likely needs to execute that many queries before it is available.

If you need to set up remote JMX so you can remotely connect jconsole, I have done this in
the redhat init script I've built -- see JMX_OPTS here:

http://wiki.apache.org/solr/ShawnHeisey#Init_script

It's never a good idea to expose Solr directly to the internet, but if you use that JMX config,
*definitely* don't expose it to the Internet.
It doesn't use any authentication.

We might need to back up a little bit and start with the problem that you are trying to figure
out, not the numbers that are being reported.

http://people.apache.org/~hossman/#xyproblem

Your original note said that you're sanity checking.  Toward that end, the only insane thing
that jumps out at me is that your max heap is
*VERY* large, and you probably don't have the proper GC tuning.

My recommendations for initial action are to use -Xmx8g on the servlet container startup and
include the GC settings you can find on the wiki pages I've given you.  It would be a very
good idea to set up remote JMX so you can use jconsole or jvisualvm remotely.

Thanks,
Shawn



Mime
View raw message