lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Solr Trunk Heap Space Issues
Date Thu, 01 Oct 2009 18:59:42 GMT
On Thu, Oct 1, 2009 at 11:41 AM, Jeff Newburn <jnewburn@zappos.com> wrote:
> I am trying to update to the newest version of solr from trunk as of May
> 5th.

Tons of changes since... including the per-segment
searching/sorting/function queries (I think).

Do you sort on any single valued fields that you also facet on?
Do you use ord() or rord() in any function queries?

Unfortunately, some of these things will take up more memory because
some things still cache FieldCache elements with the top-level reader,
while some use segment readers.  The direction is going toward all
segment readers, but we're not there yet (and won't be for 1.4).
ord() rord() will never be fixed... people need to migrate to
something else.

http://issues.apache.org/jira/browse/SOLR-1111 is the main issue for this.

If course, I've really only been talking about search related changes.
 Nothing on the indexing side should cause greater memory usage....
but perhaps the indexing side could run out of memory due to the
search side taking up more.

-Yonik
http://www.lucidimagination.com

>  I updated and compiled from trunk as of yesterday (09/30/2009).  When
> I try to do a full import I am receiving a GC heap error after changing
> nothing in the configuration files.  Why would this happen in the most
> recent versions but not in the version from a few months ago.  The stack
> trace is below.
>
> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
> finish
> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, 167353,
> ...(83 more)]} 0 35991
> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>    at java.util.Arrays.copyOfRange(Arrays.java:3209)
>    at java.lang.String.<init>(String.java:215)
>    at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>    at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>    at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>    at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>    at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>    at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
> reamHandlerBase.java:54)
>    at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
> java:131)
>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
> 38)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 241)
>    at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
>    at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
>    at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
>    at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
>    at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
>    at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
>    at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
>    at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>    at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
>    at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
>    at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
> 2080)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>    at java.lang.Thread.run(Thread.java:619)
>
> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 QTime=5265
> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>

Mime
View raw message