lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Solr Trunk Heap Space Issues
Date Tue, 06 Oct 2009 16:26:57 GMT
TestIndexingPerformance?
What the heck... that's not even multi-threaded!

-Yonik
http://www.lucidimagination.com



On Tue, Oct 6, 2009 at 12:17 PM, Mark Miller <markrmiller@gmail.com> wrote:
> Darnit - didn't finish that email. This is after running your old short
> doc perf test for 10,000 iterations. You see the same thing with 1000
> iterations but much less pronounced eg gettin' worse with more iterations.
>
> Mark Miller wrote:
>> A little before and after. The before is around may 5th'is - the after
>> is trunk.
>>
>> http://myhardshadow.com/memanalysis/before.png
>> http://myhardshadow.com/memanalysis/after.png
>>
>> Mark Miller wrote:
>>
>>> Took a peak at the checkout around the time he says he's using.
>>>
>>> CharTokenizer appears to be holding onto much large char[] arrays now
>>> than before. Same with snowball.Among - used to be almost nothing, now
>>> its largio.
>>>
>>> The new TokenStream stuff appears to be clinging. Needs to find some
>>> inner peace.
>>>
>>> Yonik Seeley wrote:
>>>
>>>
>>>> On Mon, Oct 5, 2009 at 4:54 PM, Jeff Newburn <jnewburn@zappos.com>
wrote:
>>>>
>>>>
>>>>
>>>>> Ok we have done some more testing on this issue.  When I only have the
1
>>>>> core the reindex completes fine.  However, when I added a second core
with
>>>>> no documents it runs out of heap again.  This time the heap was 322Mb
of
>>>>> LRUCache.  The 1 query that warms returns exactly 2 documents so I have
no
>>>>> idea where the LRUCache is getting its information or what is even in
there.
>>>>>
>>>>>
>>>>>
>>>> I guess the obvious thing to check would be the custom search component.
>>>> Does it access documents?  I don't see how else the document cache
>>>> could self populate with so many entries (assuming it is the document
>>>> cache again).
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> From: Yonik Seeley <yonik@lucidimagination.com>
>>>>>> Reply-To: <solr-user@lucene.apache.org>
>>>>>> Date: Mon, 5 Oct 2009 13:32:32 -0400
>>>>>> To: <solr-user@lucene.apache.org>
>>>>>> Subject: Re: Solr Trunk Heap Space Issues
>>>>>>
>>>>>> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewburn@zappos.com>
wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Ok I have eliminated all queries for warming and am still getting
the heap
>>>>>>> space dump.  Any ideas at this point what could be wrong?  This
seems like a
>>>>>>> huge increase in memory to go from indexing without issues to
not being able
>>>>>>> to even with warming off.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Do you have any custom Analyzers, Tokenizers, TokenFilters?
>>>>>> Another change is that token streams are reused by caching in a
>>>>>> thread-local, so every thread in your server could potentially have
a
>>>>>> copy of an analysis chain (token stream) per field that you have
used.
>>>>>>  This normally shouldn't be an issue since these will be small.
 Also,
>>>>>> how many unique fields do you have?
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Jeff Newburn
>>>>>>> Software Engineer, Zappos.com
>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> From: Jeff Newburn <jnewburn@zappos.com>
>>>>>>>> Reply-To: <solr-user@lucene.apache.org>
>>>>>>>> Date: Thu, 01 Oct 2009 08:41:18 -0700
>>>>>>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>>>>>>>> Subject: Solr Trunk Heap Space Issues
>>>>>>>>
>>>>>>>> I am trying to update to the newest version of solr from
trunk as of May
>>>>>>>> 5th.  I updated and compiled from trunk as of yesterday
(09/30/2009).  When
>>>>>>>> I try to do a full import I am receiving a GC heap error
after changing
>>>>>>>> nothing in the configuration files.  Why would this happen
in the most
>>>>>>>> recent versions but not in the version from a few months
ago.  The stack
>>>>>>>> trace is below.
>>>>>>>>
>>>>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>>>>>>> finish
>>>>>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097,
167316, 167353,
>>>>>>>> ...(83 more)]} 0 35991
>>>>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException
log
>>>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>     at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>>>>>>     at java.lang.String.<init>(String.java:215)
>>>>>>>>     at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>>>>>>     at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>>>>>>     at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>>>>>>     at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>>>>>>     at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>>>>>>     at
>>>>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>>>>>>> reamHandlerBase.java:54)
>>>>>>>>     at
>>>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>>>>>>> java:131)
>>>>>>>>     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>>>>>>     at
>>>>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>>>>>>> 38)
>>>>>>>>     at
>>>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>>>>>>> 241)
>>>>>>>>     at
>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>>>>>>> FilterChain.java:235)
>>>>>>>>     at
>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>>>>>>> ain.java:206)
>>>>>>>>     at
>>>>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>>>>>>> va:233)
>>>>>>>>     at
>>>>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>>>>>>> va:175)
>>>>>>>>     at
>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>>>>>>> )
>>>>>>>>     at
>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>>>>>>> )
>>>>>>>>     at
>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>>>>>>> :109)
>>>>>>>>     at
>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>>>>>     at
>>>>>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>>>>>>> 879)
>>>>>>>>     at
>>>>>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>>>>>>> ttp11NioProtocol.java:719)
>>>>>>>>     at
>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>>>>>>> 2080)
>>>>>>>>     at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>>>>>>> va:886)
>>>>>>>>     at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>>>>>>> 08)
>>>>>>>>     at java.lang.Thread.run(Thread.java:619)
>>>>>>>>
>>>>>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
>>>>>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500
QTime=5265
>>>>>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException
log
>>>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jeff Newburn
>>>>>>>> Software Engineer, Zappos.com
>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>

Mime
View raw message