lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Solr Trunk Heap Space Issues
Date Tue, 06 Oct 2009 16:31:40 GMT
Yeah - I was wondering about that ... not sure how these guys are
stacking up ...

Yonik Seeley wrote:
> TestIndexingPerformance?
> What the heck... that's not even multi-threaded!
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Tue, Oct 6, 2009 at 12:17 PM, Mark Miller <markrmiller@gmail.com> wrote:
>   
>> Darnit - didn't finish that email. This is after running your old short
>> doc perf test for 10,000 iterations. You see the same thing with 1000
>> iterations but much less pronounced eg gettin' worse with more iterations.
>>
>> Mark Miller wrote:
>>     
>>> A little before and after. The before is around may 5th'is - the after
>>> is trunk.
>>>
>>> http://myhardshadow.com/memanalysis/before.png
>>> http://myhardshadow.com/memanalysis/after.png
>>>
>>> Mark Miller wrote:
>>>
>>>       
>>>> Took a peak at the checkout around the time he says he's using.
>>>>
>>>> CharTokenizer appears to be holding onto much large char[] arrays now
>>>> than before. Same with snowball.Among - used to be almost nothing, now
>>>> its largio.
>>>>
>>>> The new TokenStream stuff appears to be clinging. Needs to find some
>>>> inner peace.
>>>>
>>>> Yonik Seeley wrote:
>>>>
>>>>
>>>>         
>>>>> On Mon, Oct 5, 2009 at 4:54 PM, Jeff Newburn <jnewburn@zappos.com>
wrote:
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> Ok we have done some more testing on this issue.  When I only have
the 1
>>>>>> core the reindex completes fine.  However, when I added a second
core with
>>>>>> no documents it runs out of heap again.  This time the heap was 322Mb
of
>>>>>> LRUCache.  The 1 query that warms returns exactly 2 documents so
I have no
>>>>>> idea where the LRUCache is getting its information or what is even
in there.
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> I guess the obvious thing to check would be the custom search component.
>>>>> Does it access documents?  I don't see how else the document cache
>>>>> could self populate with so many entries (assuming it is the document
>>>>> cache again).
>>>>>
>>>>> -Yonik
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> --
>>>>>> Jeff Newburn
>>>>>> Software Engineer, Zappos.com
>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> From: Yonik Seeley <yonik@lucidimagination.com>
>>>>>>> Reply-To: <solr-user@lucene.apache.org>
>>>>>>> Date: Mon, 5 Oct 2009 13:32:32 -0400
>>>>>>> To: <solr-user@lucene.apache.org>
>>>>>>> Subject: Re: Solr Trunk Heap Space Issues
>>>>>>>
>>>>>>> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewburn@zappos.com>
wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Ok I have eliminated all queries for warming and am still
getting the heap
>>>>>>>> space dump.  Any ideas at this point what could be wrong?
 This seems like a
>>>>>>>> huge increase in memory to go from indexing without issues
to not being able
>>>>>>>> to even with warming off.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Do you have any custom Analyzers, Tokenizers, TokenFilters?
>>>>>>> Another change is that token streams are reused by caching in
a
>>>>>>> thread-local, so every thread in your server could potentially
have a
>>>>>>> copy of an analysis chain (token stream) per field that you have
used.
>>>>>>>  This normally shouldn't be an issue since these will be small.
 Also,
>>>>>>> how many unique fields do you have?
>>>>>>>
>>>>>>> -Yonik
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Jeff Newburn
>>>>>>>> Software Engineer, Zappos.com
>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> From: Jeff Newburn <jnewburn@zappos.com>
>>>>>>>>> Reply-To: <solr-user@lucene.apache.org>
>>>>>>>>> Date: Thu, 01 Oct 2009 08:41:18 -0700
>>>>>>>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
>>>>>>>>> Subject: Solr Trunk Heap Space Issues
>>>>>>>>>
>>>>>>>>> I am trying to update to the newest version of solr from
trunk as of May
>>>>>>>>> 5th.  I updated and compiled from trunk as of yesterday
(09/30/2009).  When
>>>>>>>>> I try to do a full import I am receiving a GC heap error
after changing
>>>>>>>>> nothing in the configuration files.  Why would this happen
in the most
>>>>>>>>> recent versions but not in the version from a few months
ago.  The stack
>>>>>>>>> trace is below.
>>>>>>>>>
>>>>>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>>>>>>>> finish
>>>>>>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097,
167316, 167353,
>>>>>>>>> ...(83 more)]} 0 35991
>>>>>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException
log
>>>>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit
exceeded
>>>>>>>>>     at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>>>>>>>     at java.lang.String.<init>(String.java:215)
>>>>>>>>>     at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>>>>>>>     at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>>>>>>>     at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>>>>>>>     at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>>>>>>>     at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>>>>>>>     at
>>>>>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>>>>>>>> reamHandlerBase.java:54)
>>>>>>>>>     at
>>>>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>>>>>>>> java:131)
>>>>>>>>>     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>>>>>>>     at
>>>>>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>>>>>>>> 38)
>>>>>>>>>     at
>>>>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>>>>>>>> 241)
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>>>>>>>> FilterChain.java:235)
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>>>>>>>> ain.java:206)
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>>>>>>>> va:233)
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>>>>>>>> va:175)
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>>>>>>>> )
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>>>>>>>> )
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>>>>>>>> :109)
>>>>>>>>>     at
>>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>>>>>>     at
>>>>>>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>>>>>>>> 879)
>>>>>>>>>     at
>>>>>>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>>>>>>>> ttp11NioProtocol.java:719)
>>>>>>>>>     at
>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>>>>>>>> 2080)
>>>>>>>>>     at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>>>>>>>> va:886)
>>>>>>>>>     at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>>>>>>>> 08)
>>>>>>>>>     at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>
>>>>>>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore
execute
>>>>>>>>> INFO: [zeta-main] webapp=/solr path=/update params={}
status=500 QTime=5265
>>>>>>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException
log
>>>>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit
exceeded
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Newburn
>>>>>>>>> Software Engineer, Zappos.com
>>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>
>>>>>>>>                 
>>>>>>
>>>>>>             
>>>>         
>>>
>>>       
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>>     


-- 
- Mark

http://www.lucidimagination.com




Mime
View raw message