lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: OOM on uninvert field request
Date Wed, 30 Jun 2010 03:42:25 GMT
Yes, it is better to use ints for ids than strings. Also, the Trie int
fields have a compressed format that may cut the storage needs even
more. 8m * 4 = 32mb, times "a few hundred", we'll say 300, is 900mb of
IDs.  I don't know how these fields are stored, but if they are
separate objects we've blown up to several gigs (per-object overheads
are surprising).

4G is probably not enough for what you want. If you watch the total
memory with 'top' and hit it with different queries, you will get a
stronger sense of how much memory your use cases need.

On Tue, Jun 29, 2010 at 4:32 PM, Robert Petersen <robertpe@buy.com> wrote:
> Hello I am trying to find the right max and min settings for Java 1.6 on 20GB index with
8 million docs, running 1.6_018 JVM with solr 1.4, and am currently have java set to an even
4GB (export JAVA_OPTS="-Xmx4096m -Xms4096m") for both min and max which is doing pretty well
but occasionally still getting the below OOM errors.  We're running on dual quad core xeons
with 16GB memory installed.  I've been getting the below OOM exceptions still though.
>
> Is the memsize mentioned in the INFO for the uninvert in bytes?  Ie is memSize=29604020
mean 29MB?  We have a few hundred of these fields and they contain ints used as IDs, and
so I guess could they eat all the memory to uninvert them all after we apply load and enough
queries are performed.  Does the field type matter, would int be better than string if these
are lookup ids sparsely populated across the index?  BTW these are used for faceting and
filtering only.
>
>                <dynamicField name="*_contentAttributeToken"  type="string"
 indexed="true" multiValued="true"   stored="true" required="false"/>
>
> Jun 29, 2010 3:54:50 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field {field=768_contentAttributeToken,memSize=29604014,tindexSize=50,time=1841,phase1=1824,nTerms=1,bigTerms=0,termInstances=18,uses=0}
> Jun 29, 2010 3:54:52 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field {field=749_contentAttributeToken,memSize=29604020,tindexSize=56,time=1847,phase1=1829,nTerms=143,bigTerms=0,termInstances=951,uses=0}
> Jun 29, 2010 3:54:59 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>        at org.apache.solr.request.UnInvertedField.uninvert(UnInvertedField.java:191)
>        at org.apache.solr.request.UnInvertedField.<init>(UnInvertedField.java:178)
>        at org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:839)
>        at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:250)
>        at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:283)
>        at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:166)
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message