lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Solr Trunk Heap Space Issues
Date Tue, 06 Oct 2009 15:59:08 GMT
Took a peak at the checkout around the time he says he's using.

CharTokenizer appears to be holding onto much large char[] arrays now
than before. Same with snowball.Among - used to be almost nothing, now
its largio.

The new TokenStream stuff appears to be clinging. Needs to find some
inner peace.

Yonik Seeley wrote:
> On Mon, Oct 5, 2009 at 4:54 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
>   
>> Ok we have done some more testing on this issue.  When I only have the 1
>> core the reindex completes fine.  However, when I added a second core with
>> no documents it runs out of heap again.  This time the heap was 322Mb of
>> LRUCache.  The 1 query that warms returns exactly 2 documents so I have no
>> idea where the LRUCache is getting its information or what is even in there.
>>     
>
> I guess the obvious thing to check would be the custom search component.
> Does it access documents?  I don't see how else the document cache
> could self populate with so many entries (assuming it is the document
> cache again).
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
>
>   
>> --
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> jnewburn@zappos.com - 702-943-7562
>>
>>
>>     
>>> From: Yonik Seeley <yonik@lucidimagination.com>
>>> Reply-To: <solr-user@lucene.apache.org>
>>> Date: Mon, 5 Oct 2009 13:32:32 -0400
>>> To: <solr-user@lucene.apache.org>
>>> Subject: Re: Solr Trunk Heap Space Issues
>>>
>>> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewburn@zappos.com> wrote:
>>>       
>>>> Ok I have eliminated all queries for warming and am still getting the heap
>>>> space dump.  Any ideas at this point what could be wrong?  This seems like
a
>>>> huge increase in memory to go from indexing without issues to not being able
>>>> to even with warming off.
>>>>         
>>> Do you have any custom Analyzers, Tokenizers, TokenFilters?
>>> Another change is that token streams are reused by caching in a
>>> thread-local, so every thread in your server could potentially have a
>>> copy of an analysis chain (token stream) per field that you have used.
>>>  This normally shouldn't be an issue since these will be small.  Also,
>>> how many unique fields do you have?
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>       
>>>> Jeff Newburn
>>>> Software Engineer, Zappos.com
>>>> jnewburn@zappos.com - 702-943-7562
>>>>
>>>>
>>>>         
>>>>> From: Jeff Newburn <jnewburn@zappos.com>
>>>>> Reply-To: <solr-user@lucene.apache.org>
>>>>> Date: Thu, 01 Oct 2009 08:41:18 -0700
>>>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>>>>> Subject: Solr Trunk Heap Space Issues
>>>>>
>>>>> I am trying to update to the newest version of solr from trunk as of
May
>>>>> 5th.  I updated and compiled from trunk as of yesterday (09/30/2009).
 When
>>>>> I try to do a full import I am receiving a GC heap error after changing
>>>>> nothing in the configuration files.  Why would this happen in the most
>>>>> recent versions but not in the version from a few months ago.  The stack
>>>>> trace is below.
>>>>>
>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>>>> finish
>>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, 167353,
>>>>> ...(83 more)]} 0 35991
>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log
>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>     at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>>>     at java.lang.String.<init>(String.java:215)
>>>>>     at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>>>     at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>>>     at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>>>     at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>>>     at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>>>     at
>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>>>> reamHandlerBase.java:54)
>>>>>     at
>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>>>> java:131)
>>>>>     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>>>     at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>>>> 38)
>>>>>     at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>>>> 241)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>>>> FilterChain.java:235)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>>>> ain.java:206)
>>>>>     at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>>>> va:233)
>>>>>     at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>>>> va:175)
>>>>>     at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>>>> )
>>>>>     at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>>>> )
>>>>>     at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>>>> :109)
>>>>>     at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>>     at
>>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>>>> 879)
>>>>>     at
>>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>>>> ttp11NioProtocol.java:719)
>>>>>     at
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>>>> 2080)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>>>> va:886)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>>>> 08)
>>>>>     at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
>>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 QTime=5265
>>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log
>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>>           
>>>>         
>>     


-- 
- Mark

http://www.lucidimagination.com




Mime
View raw message