lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: Interesting OutOfMemoryError on a 170M index
Date Wed, 13 Jan 2010 22:28:16 GMT
The time in autocommit is in milliseconds. You are committing every
second while indexing.  This then causes a build-up of sucessive index
readers that absorb each commit, which is probably the out-of-memory.

On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick
<nick.minutello@credit-suisse.com> wrote:
>
> Hi,
>
> I have a bit of an interesting OutOfMemoryError that I'm trying to
> figure out.
>
> My client & Solr server are running in the same JVM (for deployment
> simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied
> code for the http-based client interface. Solr 1.3.0.
>
> My app is adding about 20,000 documents per minute to the index - one at
> a time (it is listening to an event stream and for every event, it adds
> a new document to the index).
> The size of the documents, however, is tiny - the total index growth is
> only about 170M (after about 1 hr and the OutOfMemoryError)
> At this point, there is zero querying happening - just updates to the
> index (only adding documents, no updates or deletes)
> After about an hour or so, my JVM runs out of heap space - and if I look
> at the memory utilisation over time, it looks like a classic memory
> leak. It slowly ramps up until we end up with constant FULL GC's and
> eventual OOME. Max heap space is 512M.
>
> In Solr, I'm using autocommit (to buffer the updates)
>        <autoCommit>
>          <maxDocs>10000</maxDocs>
>          <maxTime>1000</maxTime>
>        </autoCommit>
>
> (Aside: Now, I'm not sure if I am meant to call commit or not on the
> client SolrServer class if I am using autocommit - but as it turns out,
> I get OOME whether I do that or not)
>
> Any suggestions/advice of quick things to check before I dust off the
> profiler?
>
> Thanks in advance.
>
> Cheers,
> Nick
>
> ===============================================================================
>  Please access the attached hyperlink for an important electronic communications disclaimer:
>  http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>  ===============================================================================
>
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message