lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: OutOfMemoryError on small search in large, simple index
Date Mon, 07 Jan 2008 22:20:30 GMT
Please post your results, Lars!

Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
From: Lars Clausen <lc@statsbiblioteket.dk>
To: java-user@lucene.apache.org
Sent: Monday, January 7, 2008 5:00:54 AM
Subject: Re: OutOfMemoryError on small search in large, simple index


On Tue, 2008-01-01 at 23:38 -0800, Chris Hostetter wrote:
> : On Wed, 2007-12-12 at 11:37 +0100, Lars Clausen wrote:
> 
> : Seems there's a reason we still use all this memory:
> : SegmentReader.fakeNorms() creates the full-size array for us
 anyway, so
> : the memory usage cannot be avoided as long as somebody asks for the
> : norms array at any point.  The solution is starting to look like a
 patch
> 
> Minor (late) nit:  fakeNorms will reuse the same array for every
 field 
> that doesn't have norms -- so while omiting norms from only one field
 may 
> not save you any RAM, omiting norms for 10 fields will save you the
 RAM 
> used by 9 of those fields.  (correct?)

Doesn't appear to be the case in our test.  We had two fields with
norms, omitting saved only about 4MB for 50 million entries.  I'm going
to try the patch that's attached to a bug report somewhere, which fakes
the norms without making an array for it.

-Lars



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message