lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Solr Trunk Heap Space Issues
Date Mon, 05 Oct 2009 21:02:44 GMT
On Mon, Oct 5, 2009 at 4:54 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
> Ok we have done some more testing on this issue.  When I only have the 1
> core the reindex completes fine.  However, when I added a second core with
> no documents it runs out of heap again.  This time the heap was 322Mb of
> LRUCache.  The 1 query that warms returns exactly 2 documents so I have no
> idea where the LRUCache is getting its information or what is even in there.

I guess the obvious thing to check would be the custom search component.
Does it access documents?  I don't see how else the document cache
could self populate with so many entries (assuming it is the document
cache again).

-Yonik
http://www.lucidimagination.com




>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Yonik Seeley <yonik@lucidimagination.com>
>> Reply-To: <solr-user@lucene.apache.org>
>> Date: Mon, 5 Oct 2009 13:32:32 -0400
>> To: <solr-user@lucene.apache.org>
>> Subject: Re: Solr Trunk Heap Space Issues
>>
>> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
>>> Ok I have eliminated all queries for warming and am still getting the heap
>>> space dump.  Any ideas at this point what could be wrong?  This seems like
a
>>> huge increase in memory to go from indexing without issues to not being able
>>> to even with warming off.
>>
>> Do you have any custom Analyzers, Tokenizers, TokenFilters?
>> Another change is that token streams are reused by caching in a
>> thread-local, so every thread in your server could potentially have a
>> copy of an analysis chain (token stream) per field that you have used.
>>  This normally shouldn't be an issue since these will be small.  Also,
>> how many unique fields do you have?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>>
>>>
>>>> From: Jeff Newburn <jnewburn@zappos.com>
>>>> Reply-To: <solr-user@lucene.apache.org>
>>>> Date: Thu, 01 Oct 2009 08:41:18 -0700
>>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>>>> Subject: Solr Trunk Heap Space Issues
>>>>
>>>> I am trying to update to the newest version of solr from trunk as of May
>>>> 5th.  I updated and compiled from trunk as of yesterday (09/30/2009).  When
>>>> I try to do a full import I am receiving a GC heap error after changing
>>>> nothing in the configuration files.  Why would this happen in the most
>>>> recent versions but not in the version from a few months ago.  The stack
>>>> trace is below.
>>>>
>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>>> finish
>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, 167353,
>>>> ...(83 more)]} 0 35991
>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log
>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>     at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>>     at java.lang.String.<init>(String.java:215)
>>>>     at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>>     at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>>     at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>>     at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>>     at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>>     at
>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>>> reamHandlerBase.java:54)
>>>>     at
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>>> java:131)
>>>>     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>>     at
>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>>> 38)
>>>>     at
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>>> 241)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>>> FilterChain.java:235)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>>> ain.java:206)
>>>>     at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>>> va:233)
>>>>     at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>>> va:175)
>>>>     at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>>> )
>>>>     at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>>> )
>>>>     at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>>> :109)
>>>>     at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>     at
>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>>> 879)
>>>>     at
>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>>> ttp11NioProtocol.java:719)
>>>>     at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>>> 2080)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>>> va:886)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>>> 08)
>>>>     at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 QTime=5265
>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log
>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>
>>>> --
>>>> Jeff Newburn
>>>> Software Engineer, Zappos.com
>>>> jnewburn@zappos.com - 702-943-7562
>>>>
>>>
>>>
>
>

Mime
View raw message