lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Anand.Ni...@rbs.com>
Subject RE: OutOfMemoryError coming from TermVectorsReader
Date Fri, 23 Sep 2011 03:56:02 GMT
Hi,

I am trying to index application log files and some database tables. Size of the log files
range from 1 MB to 100 MB. Database tables also have few thousands of rows.

I have used termvector highlighter for the content of the log files as mentioned below:

Heap size : 10 GB 
OS: Linux, 64 bit
Solr version : 3.4.0

Thanks & Regards
Anand



Anand Nigam
RBS Global Banking & Markets
Office: +91 124 492 5506   

-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com] 
Sent: 19 September 2011 16:52
To: solr-user@lucene.apache.org
Subject: Re: OutOfMemoryError coming from TermVectorsReader

Please include information about your heap size, (and other Java command line arguments) as
well a platform OS (version, swap size, etc), Java version, underlying hardware (RAM, etc)
for us to better help you.

>From the information you have given, increasing your heap size should help.

Thanks,
Glen

http://zzzoot.blogspot.com/


On Mon, Sep 19, 2011 at 1:34 AM,  <Anand.Nigam@rbs.com> wrote:
> Hi,
>
> I am new to solr. I an trying to index text documents of large size. On searching from
indexed documents I am getting following OutOfMemoryError. Please help me in resolving this
issue.
>
> The field which stores file content is configured in schema.xml as below:
>
>
> <field name="Content" type="text_token" indexed="true" stored="true" 
> omitNorms="true" termVectors="true" termPositions="true" 
> termOffsets="true" />
>
> and Highlighting is configured as below:
>
>
> <str name="hl">on</str>
>
> <str name="hl.fl">${all.fields.list}</str>
>
> <str name="f.Content.hl.fragsize">500</str>
>
> <str name="f.Content.hl.useFastVectorHighlighter">true</str>
>
>
>
> 2011-09-16 09:38:45.763 [http-thread-pool-9091(5)] ERROR - 
> java.lang.OutOfMemoryError: Java heap space
>        at 
> org.apache.lucene.index.TermVectorsReader.readTermVector(TermVectorsRe
> ader.java:503)
>        at 
> org.apache.lucene.index.TermVectorsReader.get(TermVectorsReader.java:2
> 63)
>        at 
> org.apache.lucene.index.TermVectorsReader.get(TermVectorsReader.java:2
> 84)
>        at 
> org.apache.lucene.index.SegmentReader.getTermFreqVector(SegmentReader.
> java:759)
>        at 
> org.apache.lucene.index.DirectoryReader.getTermFreqVector(DirectoryRea
> der.java:510)
>        at 
> org.apache.solr.search.SolrIndexReader.getTermFreqVector(SolrIndexRead
> er.java:234)
>        at 
> org.apache.lucene.search.vectorhighlight.FieldTermStack.<init>(FieldTe
> rmStack.java:83)
>        at 
> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFiel
> dFragList(FastVectorHighlighter.java:175)
>        at 
> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getBest
> Fragments(FastVectorHighlighter.java:166)
>        at 
> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByFastV
> ectorHighlighter(DefaultSolrHighlighter.java:509)
>        at 
> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(Defaul
> tSolrHighlighter.java:376)
>        at 
> org.apache.solr.handler.component.HighlightComponent.process(Highlight
> Component.java:116)
>        at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
> chHandler.java:194)
>        at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rBase.java:129)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.
> java:356)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter
> .java:252)
>        at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> cationFilterChain.java:256)
>        at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
> lterChain.java:215)
>        at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> lve.java:279)
>        at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> lve.java:175)
>        at 
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.ja
> va:655)
>        at 
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
> :595)
>        at 
> com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
>        at 
> com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessi
> onLockingStandardPipeline.java:91)
>        at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> va:162)
>        at 
> org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.ja
> va:326)
>        at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> :227)
>        at 
> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerM
> apper.java:170)
>        at 
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:82
> 2)
>        at 
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
>        at 
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013)
>
> Thanks & Regards
> Anand Nigam
> Developer
>
>
> **********************************************************************
> ************* The Royal Bank of Scotland plc. Registered in Scotland 
> No 90312.
> Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB.
> Authorised and regulated by the Financial Services Authority. The 
> Royal Bank of Scotland N.V. is authorised and regulated by the De 
> Nederlandsche Bank and has its seat at Amsterdam, the Netherlands, and 
> is registered in the Commercial Register under number 33002587. 
> Registered Office: Gustav Mahlerlaan 350, Amsterdam, The Netherlands. 
> The Royal Bank of Scotland N.V. and The Royal Bank of Scotland plc are 
> authorised to act as agent for each other in certain jurisdictions.
>
> This e-mail message is confidential and for use by the addressee only.
> If the message is received by anyone other than the addressee, please 
> return the message to the sender by replying to it and then delete the 
> message from your computer. Internet e-mails are not necessarily 
> secure. The Royal Bank of Scotland plc and The Royal Bank of Scotland 
> N.V. including its affiliates ("RBS group") does not accept 
> responsibility for changes made to this message after it was sent. For 
> the protection of RBS group and its clients and customers, and in 
> compliance with regulatory requirements, the contents of both incoming 
> and outgoing e-mail communications, which could include proprietary 
> information and Non-Public Personal Information, may be read by 
> authorised persons within RBS group other than the intended recipient(s).
>
> Whilst all reasonable care has been taken to avoid the transmission of 
> viruses, it is the responsibility of the recipient to ensure that the 
> onward transmission, opening or use of this message and any 
> attachments will not adversely affect its systems or data. No 
> responsibility is accepted by the RBS group in this regard and the 
> recipient should carry out such virus and other checks as it considers appropriate.
>
> Visit our website at www.rbs.com
>
> **********************************************************************
> *************
>



--
-

-

Mime
View raw message