lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan McKinley <ryan...@gmail.com>
Subject Re: Interesting OutOfMemoryError on a 170M index
Date Wed, 13 Jan 2010 23:16:08 GMT

On Jan 13, 2010, at 5:34 PM, Minutello, Nick wrote:

> Agreed, commit every second.

Do you need the index to be updated this often?  Are you reading from  
it every second?  and need results that are that "fresh"

If not, i imagine increasing the auto-commit time to 1min or even 10  
secs would help some.

Re, calling commit from the client with auto-commit...  if you are  
using auto-commit, you should not call commit from the client

ryan



>
> Assuming I understand what you're saying correctly:
> There shouldn't be any index readers - as at this point, just  
> writing to the index.
> Did I understand correctly what you meant?
>
> -Nick
>
> -----Original Message-----
> From: Lance Norskog [mailto:goksron@gmail.com]
> Sent: 13 January 2010 22:28
> To: solr-user@lucene.apache.org
> Subject: Re: Interesting OutOfMemoryError on a 170M index
>
> The time in autocommit is in milliseconds. You are committing every  
> second while indexing.  This then causes a build-up of sucessive  
> index readers that absorb each commit, which is probably the out-of- 
> memory.
>
> On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick <nick.minutello@credit-suisse.com

> > wrote:
>>
>> Hi,
>>
>> I have a bit of an interesting OutOfMemoryError that I'm trying to
>> figure out.
>>
>> My client & Solr server are running in the same JVM (for deployment
>> simplicity). FWIW, I'm using Jetty to host Solr. I'm using the
>> supplied code for the http-based client interface. Solr 1.3.0.
>>
>> My app is adding about 20,000 documents per minute to the index - one
>> at a time (it is listening to an event stream and for every event, it
>> adds a new document to the index).
>> The size of the documents, however, is tiny - the total index growth
>> is only about 170M (after about 1 hr and the OutOfMemoryError) At  
>> this
>> point, there is zero querying happening - just updates to the index
>> (only adding documents, no updates or deletes) After about an hour or
>> so, my JVM runs out of heap space - and if I look at the memory
>> utilisation over time, it looks like a classic memory leak. It slowly
>> ramps up until we end up with constant FULL GC's and eventual OOME.
>> Max heap space is 512M.
>>
>> In Solr, I'm using autocommit (to buffer the updates)
>>        <autoCommit>
>>          <maxDocs>10000</maxDocs>
>>          <maxTime>1000</maxTime>
>>        </autoCommit>
>>
>> (Aside: Now, I'm not sure if I am meant to call commit or not on the
>> client SolrServer class if I am using autocommit - but as it turns
>> out, I get OOME whether I do that or not)
>>
>> Any suggestions/advice of quick things to check before I dust off the
>> profiler?
>>
>> Thanks in advance.
>>
>> Cheers,
>> Nick
>>
>> = 
>> =====================================================================
>> =========
>>  Please access the attached hyperlink for an important electronic  
>> communications disclaimer:
>>  http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>
>> = 
>> =====================================================================
>> =========
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
> = 
> = 
> = 
> = 
> = 
> = 
> = 
> = 
> = 
> ======================================================================
> Please access the attached hyperlink for an important electronic  
> communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> = 
> = 
> = 
> = 
> = 
> = 
> = 
> = 
> = 
> ======================================================================
>


Mime
View raw message