lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Solr Trunk Heap Space Issues
Date Thu, 01 Oct 2009 20:05:19 GMT
On Thu, Oct 1, 2009 at 3:37 PM, Mark Miller <markrmiller@gmail.com> wrote:
> Still interested in seeing his field sanity output to see whats possibly
> being doubled.

Strangely enough, I'm having a hard time seeing caching at the different levels.
I mad a multi-segment index (2 segments), and then did a sort and facet:
http://localhost:8983/solr/select?q=*:*&sort=popularity%20desc&facet=true&facet.field=popularity

Seems like that should do it, but the statistics fieldCache section
shows only 2 entries.
  entries_count : 2
entry#0 : 'org.apache.lucene.index.CompoundFileReader$CSIndexInput@5b38d7'=>'popularity',int,org.apache.lucene.search.FieldCache.NUMERIC_UTILS_INT_PARSER=>[I#949587
(size =~ 92 bytes)
entry#1 : 'org.apache.lucene.index.CompoundFileReader$CSIndexInput@1582a7c'=>'popularity',int,org.apache.lucene.search.FieldCache.NUMERIC_UTILS_INT_PARSER=>[I#3534544
(size =~ 28 bytes)
insanity_count : 0

Investigating further...

-Yonik
http://www.lucidimagination.com

> Yonik Seeley wrote:
>> On Thu, Oct 1, 2009 at 3:14 PM, Mark Miller <markrmiller@gmail.com> wrote:
>>
>>> bq. Tons of changes since... including the per-segment
>>> searching/sorting/function queries (I think).
>>>
>>> Yup. I actually didn't think so, because that was committed to Lucene in
>>> Feburary - but it didn't come into Solr till March 10th. March 5th just
>>> ducked it.
>>>
>>
>> Jeff said May 5th
>>
>> But it wasn't until the end of May that Solr started using Lucene's
>> new sorting facilities that worked per-segment.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Yonik Seeley wrote:
>>>
>>>> On Thu, Oct 1, 2009 at 11:41 AM, Jeff Newburn <jnewburn@zappos.com>
wrote:
>>>>
>>>>
>>>>> I am trying to update to the newest version of solr from trunk as of
May
>>>>> 5th.
>>>>>
>>>>>
>>>> Tons of changes since... including the per-segment
>>>> searching/sorting/function queries (I think).
>>>>
>>>> Do you sort on any single valued fields that you also facet on?
>>>> Do you use ord() or rord() in any function queries?
>>>>
>>>> Unfortunately, some of these things will take up more memory because
>>>> some things still cache FieldCache elements with the top-level reader,
>>>> while some use segment readers.  The direction is going toward all
>>>> segment readers, but we're not there yet (and won't be for 1.4).
>>>> ord() rord() will never be fixed... people need to migrate to
>>>> something else.
>>>>
>>>> http://issues.apache.org/jira/browse/SOLR-1111 is the main issue for this.
>>>>
>>>> If course, I've really only been talking about search related changes.
>>>>  Nothing on the indexing side should cause greater memory usage....
>>>> but perhaps the indexing side could run out of memory due to the
>>>> search side taking up more.
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>>  I updated and compiled from trunk as of yesterday (09/30/2009).  When
>>>>> I try to do a full import I am receiving a GC heap error after changing
>>>>> nothing in the configuration files.  Why would this happen in the most
>>>>> recent versions but not in the version from a few months ago.  The stack
>>>>> trace is below.
>>>>>
>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>>>> finish
>>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, 167353,
>>>>> ...(83 more)]} 0 35991
>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log
>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>    at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>>>    at java.lang.String.<init>(String.java:215)
>>>>>    at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>>>    at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>>>    at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>>>    at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>>>    at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>>>    at
>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>>>> reamHandlerBase.java:54)
>>>>>    at
>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>>>> java:131)
>>>>>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>>>    at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>>>> 38)
>>>>>    at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>>>> 241)
>>>>>    at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>>>> FilterChain.java:235)
>>>>>    at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>>>> ain.java:206)
>>>>>    at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>>>> va:233)
>>>>>    at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>>>> va:175)
>>>>>    at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>>>> )
>>>>>    at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>>>> )
>>>>>    at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>>>> :109)
>>>>>    at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>>    at
>>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>>>> 879)
>>>>>    at
>>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>>>> ttp11NioProtocol.java:719)
>>>>>    at
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>>>> 2080)
>>>>>    at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>>>> va:886)
>>>>>    at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>>>> 08)
>>>>>    at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
>>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 QTime=5265
>>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log
>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>>
>>>>>
>>>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>

Mime
View raw message