lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Newburn <jnewb...@zappos.com>
Subject Re: Solr Trunk Heap Space Issues
Date Mon, 05 Oct 2009 20:54:35 GMT
Ok we have done some more testing on this issue.  When I only have the 1
core the reindex completes fine.  However, when I added a second core with
no documents it runs out of heap again.  This time the heap was 322Mb of
LRUCache.  The 1 query that warms returns exactly 2 documents so I have no
idea where the LRUCache is getting its information or what is even in there.


-- 
Jeff Newburn
Software Engineer, Zappos.com
jnewburn@zappos.com - 702-943-7562


> From: Yonik Seeley <yonik@lucidimagination.com>
> Reply-To: <solr-user@lucene.apache.org>
> Date: Mon, 5 Oct 2009 13:32:32 -0400
> To: <solr-user@lucene.apache.org>
> Subject: Re: Solr Trunk Heap Space Issues
> 
> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
>> Ok I have eliminated all queries for warming and am still getting the heap
>> space dump.  Any ideas at this point what could be wrong?  This seems like a
>> huge increase in memory to go from indexing without issues to not being able
>> to even with warming off.
> 
> Do you have any custom Analyzers, Tokenizers, TokenFilters?
> Another change is that token streams are reused by caching in a
> thread-local, so every thread in your server could potentially have a
> copy of an analysis chain (token stream) per field that you have used.
>  This normally shouldn't be an issue since these will be small.  Also,
> how many unique fields do you have?
> 
> -Yonik
> http://www.lucidimagination.com
> 
> 
> 
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> jnewburn@zappos.com - 702-943-7562
>> 
>> 
>>> From: Jeff Newburn <jnewburn@zappos.com>
>>> Reply-To: <solr-user@lucene.apache.org>
>>> Date: Thu, 01 Oct 2009 08:41:18 -0700
>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>>> Subject: Solr Trunk Heap Space Issues
>>> 
>>> I am trying to update to the newest version of solr from trunk as of May
>>> 5th.  I updated and compiled from trunk as of yesterday (09/30/2009).  When
>>> I try to do a full import I am receiving a GC heap error after changing
>>> nothing in the configuration files.  Why would this happen in the most
>>> recent versions but not in the version from a few months ago.  The stack
>>> trace is below.
>>> 
>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>> finish
>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, 167353,
>>> ...(83 more)]} 0 35991
>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>     at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>     at java.lang.String.<init>(String.java:215)
>>>     at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>     at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>     at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>     at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>     at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>     at
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>> reamHandlerBase.java:54)
>>>     at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>> java:131)
>>>     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>     at
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>> 38)
>>>     at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>> 241)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:235)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>>     at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>> va:233)
>>>     at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>> va:175)
>>>     at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>> )
>>>     at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>> )
>>>     at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>> :109)
>>>     at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>     at
>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>> 879)
>>>     at
>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>> ttp11NioProtocol.java:719)
>>>     at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>> 2080)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:886)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 08)
>>>     at java.lang.Thread.run(Thread.java:619)
>>> 
>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 QTime=5265
>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> 
>>> --
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>> 
>> 
>> 


Mime
View raw message