lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: frequency of commit when building index from scratch
Date Wed, 26 Aug 2009 00:37:41 GMT
The latest Solr 1.4 can index 200k records in several minutes, then commit
in a few seconds. I don't know but I'm guessing it is due to Lucene
improvements. It does not use much memory doing this.

Lance

On Tue, Aug 25, 2009 at 2:43 PM, Fuad Efendi <fuad@efendi.ca> wrote:

> I do commit once a day, millions of small docs... it takes 20 minutes in
> average... why OOM? I see only reduced I/O...
>
>
> -----Original Message-----
> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> Sent: August-25-09 5:35 PM
> To: solr-user@lucene.apache.org
> Subject: Re: frequency of commit when building index from scratch
>
> On Tue, Aug 25, 2009 at 5:29 PM, Bill Au<bill.w.au@gmail.com> wrote:
> > Just curious, how often do folks commit when building their Solr/Lucene
> > index from scratch for index with millions of documents?  Should I just
> wait
> > and do a single commit at the end after adding all the documents to the
> > index?
> >
> > Bill
> >
>
> Bill in most cases you probably cannot do one large commit as you will
> hit OOM. How many documents can be uncommitted is based on the size of
> the documents. Committing every document is slow. I have done a commit
> every 10,000 mostly. Results may vary. Someone might have a better
> answer then me.
>
>
>


-- 
Lance Norskog
goksron@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message