lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: DIH OutOfMemoryError?
Date Sun, 10 Apr 2011 23:34:08 GMT
Make sure streaming is on.
Try using autoCommit in solrconfig.xml. This will push documents out
of memory onto disk at a regular interval.

On Thu, Mar 31, 2011 at 8:51 AM, Markus Jelsma
<markus.jelsma@openindex.io> wrote:
> Try splitting the files into smaller chunks. It'll help.
>
>> Hi,
>>
>> I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an
>> OutOfMemoryError!
>>
>> I've got 2048mo of RAM on this server, obviously it's not enough... How
>> much RAM is recomended for indexing big files?
>>
>> Thanks for your help
>>
>>
>> Here is the error from DIH mode verbose:
>>
>> <str name="EXCEPTION">java.lang.ClassCastException:
>> java.lang.OutOfMemoryError cannot be cast to java.lang.Exception
>>      at
>> org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139)
>>      at
>> org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237)
>>      at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java
>> :422) at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java
>> :383) at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:24
>> 2) at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
>>      at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.j
>> ava:331) at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:38
>> 9) at
>> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(Data
>> ImportHandler.java:203) at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase
>> .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>      at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:
>> 338) at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java
>> :241) at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
>> nFilterChain.java:235) at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
>> hain.java:206) at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j
>> ava:233) at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
>> ava:191) at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12
>> 7) at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10
>> 2) at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
>> a:109) at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>      at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>      at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt
>> p11Protocol.java:588) at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>      at java.lang.Thread.run(Thread.java:636)
>> </str>
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message