lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reece <liquidp...@gmail.com>
Subject Re: out of memory every time
Date Tue, 04 Mar 2008 01:20:31 GMT
Just guessing, but I'd say it has something to do with the dynamic fields...

I ran a similar operation (docs ranged from 1K to 2MB).  For the
initial indexing, I wrote a job to submit about 100,000 documents to
solr, committing after every 10 docs.  I never sent any optimize
commands.  I also used the example start.jar and didn't specify any
memory constraints.

My job ran for 3 days, and finished without any errors or memory problems.

The only difference I see is that I didn't use any dynamic fields, and
I only stored 2 fields instead of them all.

Just my $0.02
-Reece



On Mon, Mar 3, 2008 at 6:15 PM, Thorsten Scherler <thorsten@apache.org> wrote:
> On Mon, 2008-03-03 at 21:43 +0200, Justin wrote:
>  > I'm indexing a large number of documents.
>  >
>  > As a server I'm using the /solr/example/start.jar
>  >
>  > No matter how much memory I allocate it fails around 7200 documents.
>
>  How do you allocate the memory?
>
>  Something like:
>  java -Xms512M -Xmx1500M -jar start.jar
>
>  You may have a closer look as well at
>  http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html
>
>  HTH
>
>  salu2
>
>
>
>  > I am committing every 100 docs, and optimizing every 300.
>  >
>  > all of my xml's contain on doc, and can range in size from 2k to 700k.
>  >
>  > when I restart the start.jar it again reports out of memory.
>  >
>  >
>  > a sample document looks like this:
>  > <?xml version="1.0" encoding="UTF-8"?>
>  > <add>
>  >  <doc>
>  >   <field name="PK">1851</field>
>  >   <field name="ft:genes.Symbol:1851">TRAJ20</field>
>  >   <field name="ft:external_ids.SourceAccession:15531">12049</field>
>  >   <field
>  > name="ft:external_ids.SourceAccession:15532">ENSG00000211869</field>
>  >   <field name="ft:external_ids.SourceAccession:15533">28735</field>
>  >   <field name="ft:external_ids.SourceAccession:15534">HUgn28735</field>
>  >   <field name="ft:external_ids.SourceAccession:15535">TRA_</field>
>  >   <field name="ft:external_ids.SourceAccession:15536">TRAJ20</field>
>  >   <field name="ft:external_ids.SourceAccession:15537">9953837</field>
>  >   <field
>  > name="ft:external_ids.SourceAccession:15538">ENSG00000211869</field>
>  >   <field name="ft:aliases_and_descriptions.Value:9775">T cell receptor alpha
>  > joining 20</field>
>  >   <field name="ft:cytogenetic_locations.Cytoband:4909">14q11.2</field>
>  >   <field name="ft:cytogenetic_locations.Cytoband:4910">14q11</field>
>  >   <field name="ft:cytogenetic_locations.Cytoband:4911">14q11.2</field>
>  >   <field name="ft:location_extras.ContigRefseq:11806">AE000662.1</field>
>  >   <field name="ft:location_extras.ContigRefseq:11807">M94081.1</field>
>  >   <field name="ft:location_extras.ContigRefseq:11808">CH471078.2</field>
>  >   <field name="ft:location_extras.ContigRefseq:11809">NC_000014.7</field>
>  >   <field name="ft:location_extras.ContigRefseq:11810">NT_026437.11</field>
>  >   <field name="ft:location_extras.ContigRefseq:11811">NG_001332.2</field>
>  >   <field name="ft:articles.SourceAccession:192767">8188290</field>
>  >   <field name="ft:articles.Title:192767">The human T-cell receptor
>  > TCRAC/TCRDC (C alpha/C delta) region: organization,sequence, and evolution
>  > of 97.6 kb of DNA.</field>
>  >   <field name="ft:authors.AuthorName:5909">Koop B.F.</field>
>  >   <field name="ft:authors.AuthorName:6912">Rowen L.</field>
>  >   <field name="ft:authors.AuthorName:6985">Hood L.</field>
>  >   <field name="ft:authors.AuthorName:17109">Wang K.</field>
>  >   <field name="ft:authors.AuthorName:72700">Kuo C.L.</field>
>  >   <field name="ft:authors.AuthorName:84285">Seto D.</field>
>  >   <field name="ft:authors.AuthorName:166156">Lenstra J.A.</field>
>  >   <field name="ft:authors.AuthorName:216734">Howard S.</field>
>  >   <field name="ft:authors.AuthorName:285493">Shan W.</field>
>  >   <field name="ft:authors.AuthorName:346559">Deshpande P.</field>
>  >   <field name="ft:probesets.Name:6773">31311_at</field>
>  >   <field name="ft:probesets.BinaryPattern:6773">000000000000</field>
>  > </doc>
>  > </add>
>  >
>  >
>  > the schema is (in summary):
>  > <fields>
>  >    <field name="PK" type="sint" indexed="true" stored="true" required="true"
>  > multiValued="false" omitNorms="true"/>
>  >    <field name="text" type="text" indexed="true" stored="false"
>  > multiValued="true"  omitNorms="true"/>
>  >
>  >    <dynamicField name="ft:*"  type="string"    indexed="true"
>  > stored="true"  omitNorms="true"/>
>  >    <dynamicField name="st:*"  type="string"  indexed="true"  stored="true"
>  > omitNorms="true"/>
>  > </fields>
>  >
>  >
>  > <uniqueKey>PK</uniqueKey>
>  > <defaultSearchField>text</defaultSearchField>
>  > <solrQueryParser defaultOperator="OR"/>
>  >
>  > <copyField source="ft:*" dest="text"/>
>  > <copyField source="st:*" dest="text"/>
>  >
>  >
>  > and my conf is:
>  >    <useCompoundFile>false</useCompoundFile>
>  >     <mergeFactor>100</mergeFactor>
>  >     <maxBufferedDocs>900</maxBufferedDocs>
>  >     <maxMergeDocs>2147483647</maxMergeDocs>
>  >     <maxFieldLength>10000</maxFieldLength>
>  --
>  Thorsten Scherler                                 thorsten.at.apache.org
>  Open Source Java                      consulting, training and solutions
>
>

Mime
View raw message