lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Hull <char...@flax.co.uk>
Subject Re: OutOfMemoryError for PDF document upload into Solr
Date Fri, 16 Jan 2015 15:21:55 GMT
On 16/01/2015 04:02, Dan Davis wrote:
> Why re-write all the document conversion in Java ;)  Tika is very slow.   5
> GB PDF is very big.

Or you can run Tika in a separate process, or even on a separate 
machine, wrapped with something to cope if it dies due to some horrible 
input...we generally avoid document format translation within Solr and 
do it externally before feeding documents to Solr.

Charlie
>
> If you have a lot of PDF like that try pdftotext in HTML and UTF-8 output
> mode.   The HTML mode captures some meta-data that would otherwise be lost.
>
>
> If you need to go faster still, you can  also write some stuff linked
> directly against poppler library.
>
> Before you jump down by through about Tika being slow - I wrote a PDF
> indexer that ran at 36 MB/s per core.   Different indexer, all C, lots of
> getjmp/longjmp.   But fast...
>
>
>
> On Thu, Jan 15, 2015 at 1:54 PM, <Ganesh.Yadav@sungard.com> wrote:
>
>> Siegfried and Michael Thank you for your replies and help.
>>
>> -----Original Message-----
>> From: Siegfried Goeschl [mailto:sgoeschl@gmx.at]
>> Sent: Thursday, January 15, 2015 3:45 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: OutOfMemoryError for PDF document upload into Solr
>>
>> Hi Ganesh,
>>
>> you can increase the heap size but parsing a 4 GB PDF document will very
>> likely consume A LOT OF memory - I think you need to check if that large
>> PDF can be parsed at all :-)
>>
>> Cheers,
>>
>> Siegfried Goeschl
>>
>> On 14.01.15 18:04, Michael Della Bitta wrote:
>>> Yep, you'll have to increase the heap size for your Tomcat container.
>>>
>>> http://stackoverflow.com/questions/6897476/tomcat-7-how-to-set-initial
>>> -heap-size-correctly
>>>
>>> Michael Della Bitta
>>>
>>> Senior Software Engineer
>>>
>>> o: +1 646 532 3062
>>>
>>> appinions inc.
>>>
>>> “The Science of Influence Marketing”
>>>
>>> 18 East 41st Street
>>>
>>> New York, NY 10017
>>>
>>> t: @appinions <https://twitter.com/Appinions> | g+:
>>> plus.google.com/appinions
>>> <https://plus.google.com/u/0/b/112002776285509593336/11200277628550959
>>> 3336/posts>
>>> w: appinions.com <http://www.appinions.com/>
>>>
>>> On Wed, Jan 14, 2015 at 12:00 PM, <Ganesh.Yadav@sungard.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Can someone pass on the hints to get around following error? Is there
>>>> any Heap Size parameter I can set in Tomcat or in Solr webApp that
>>>> gets deployed in Solr?
>>>>
>>>> I am running Solr webapp inside Tomcat on my local machine which has
>>>> RAM of 12 GB. I have PDF document which is 4 GB max in size that
>>>> needs to be loaded into Solr
>>>>
>>>>
>>>>
>>>>
>>>> Exception in thread "http-apr-8983-exec-6" java.lang.    : Java heap
>> space
>>>>           at java.util.AbstractCollection.toArray(Unknown Source)
>>>>           at java.util.ArrayList.<init>(Unknown Source)
>>>>           at
>>>> org.apache.pdfbox.cos.COSDocument.getObjects(COSDocument.java:518)
>>>>           at
>> org.apache.pdfbox.cos.COSDocument.close(COSDocument.java:575)
>>>>           at
>> org.apache.pdfbox.pdfparser.PDFParser.parse(PDFParser.java:254)
>>>>           at
>> org.apache.pdfbox.pdmodel.PDDocument.load(PDDocument.java:1238)
>>>>           at
>> org.apache.pdfbox.pdmodel.PDDocument.load(PDDocument.java:1203)
>>>>           at
>> org.apache.tika.parser.pdf.PDFParser.parse(PDFParser.java:111)
>>>>           at
>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
>>>>           at
>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
>>>>           at
>>>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
>>>>           at
>>>>
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219)
>>>>           at
>>>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>>>           at
>>>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>>>           at
>>>>
>> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:246)
>>>>           at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
>>>>           at
>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
>>>>           at
>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
>>>>           at
>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
>>>>           at
>>>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>>>>           at
>>>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>>>>           at
>>>>
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
>>>>           at
>>>>
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
>>>>           at
>>>>
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
>>>>           at
>>>>
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>>>>           at
>>>>
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
>>>>           at
>>>>
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>>>>           at
>>>>
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
>>>>           at
>>>>
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
>>>>           at
>>>>
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
>>>>           at
>>>>
>> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2462)
>>>>           at
>>>> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoin
>>>> t.java:2451)
>>>>
>>>>
>>>> Thanks
>>>> Ganesh
>>>>
>>>>
>>>
>>
>>
>


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Mime
View raw message