lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Petersen" <>
Subject RE: OOM on uninvert field request
Date Wed, 30 Jun 2010 20:07:27 GMT
Hey so after adding those GC options, I was able to incrementally push my max (and min) memory
settings up and when we got to max=min=12GB we started looking much better!  One slave handles
all the load with no OOMs at all!  I'm watching the live tomcat log using 'tail'.  Next I
will convert that field type to (trie) int and reindex.  I'll have to start a new index from
scratch with a field type change like that so I'll have to delete the old one first on our
master... It takes us a couple days to index 15 million products (some are sets so the final
index size is only 8 million) so I don't want to do *that* too often as the slaves will be
quite stale by the time it's done!  :)

Thanks for the help!

-----Original Message-----
From: Robert Petersen [] 
Sent: Wednesday, June 30, 2010 9:49 AM
Subject: RE: OOM on uninvert field request

At and above 4GB we get those GC errors though!  Should I switch to something like this?

Recommended Options
To use i-cms in Java SE 6, use the following command line options:

-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.solr.core.SolrCore.getSearcher(
	at org.apache.solr.update.DirectUpdateHandler2.commit(
	at org.apache.solr.handler.SnapPuller.doCommit(
	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(
	... 11 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

-----Original Message-----
From: Lance Norskog [] 
Sent: Tuesday, June 29, 2010 8:42 PM
Subject: Re: OOM on uninvert field request

Yes, it is better to use ints for ids than strings. Also, the Trie int
fields have a compressed format that may cut the storage needs even
more. 8m * 4 = 32mb, times "a few hundred", we'll say 300, is 900mb of
IDs.  I don't know how these fields are stored, but if they are
separate objects we've blown up to several gigs (per-object overheads
are surprising).

4G is probably not enough for what you want. If you watch the total
memory with 'top' and hit it with different queries, you will get a
stronger sense of how much memory your use cases need.

On Tue, Jun 29, 2010 at 4:32 PM, Robert Petersen <> wrote:
> Hello I am trying to find the right max and min settings for Java 1.6 on 20GB index with
8 million docs, running 1.6_018 JVM with solr 1.4, and am currently have java set to an even
4GB (export JAVA_OPTS="-Xmx4096m -Xms4096m") for both min and max which is doing pretty well
but occasionally still getting the below OOM errors.  We're running on dual quad core xeons
with 16GB memory installed.  I've been getting the below OOM exceptions still though.
> Is the memsize mentioned in the INFO for the uninvert in bytes?  Ie is memSize=29604020
mean 29MB?  We have a few hundred of these fields and they contain ints used as IDs, and
so I guess could they eat all the memory to uninvert them all after we apply load and enough
queries are performed.  Does the field type matter, would int be better than string if these
are lookup ids sparsely populated across the index?  BTW these are used for faceting and
filtering only.
>                <dynamicField name="*_contentAttributeToken"  type="string"
 indexed="true" multiValued="true"   stored="true" required="false"/>
> Jun 29, 2010 3:54:50 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field {field=768_contentAttributeToken,memSize=29604014,tindexSize=50,time=1841,phase1=1824,nTerms=1,bigTerms=0,termInstances=18,uses=0}
> Jun 29, 2010 3:54:52 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field {field=749_contentAttributeToken,memSize=29604020,tindexSize=56,time=1847,phase1=1829,nTerms=143,bigTerms=0,termInstances=951,uses=0}
> Jun 29, 2010 3:54:59 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>        at org.apache.solr.request.UnInvertedField.uninvert(
>        at org.apache.solr.request.UnInvertedField.<init>(
>        at org.apache.solr.request.UnInvertedField.getUnInvertedField(
>        at org.apache.solr.request.SimpleFacets.getTermCounts(
>        at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(
>        at org.apache.solr.request.SimpleFacets.getFacetCounts(

Lance Norskog
View raw message