lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: OutOfMemory error while sorting
Date Wed, 14 Jun 2006 18:19:41 GMT

This is a fairly typical Lucene issue (ie: not specific to Solr)...

Sorting on a field requires building a FieldCache for every document --
regardless of how many documents match your query.  This cache is reused
for all searches thta sort on that field.

For things like Integers and Floats, the size of the FieldCache is one
item (int/float, etc) per document.  for Strings, the size is one int
per document, plus the total of every unique string field value.

This is why sorting on some String fields use more memory then other
String fields -- it all depends on hoe heterogenous the values in that
field are.  A field that only contains 4 unique values takes up a lot less
room then a field where every document has a different value.

In the end, there isn't much you can do about this except allocate more
memory to your JVM -- One option you do have in Solr is to tune other
parameters in Solr so that more of the memory you already have allocated
to the JVM is available for sorting.  (ie: making your filterCaches
smaller for example)

Off the top of my head, i don't remember if omiting norms for fields
reduces the amount of resident memory needed by the index, or just the on
disk size, but you might wnat to try that also if there are fields you
know you don't need norms for (a String field you sort on is a good bet,
since you probably don't search on it, and even if you do the length is
always going to be 1)


: I have a new problem with OutOfMemory errors.
: As I reported before, we have an index with more than 10 million
: documents and 23 fields. Recently I added a new field which we will only
: use for sorting purposes (by "adding" I mean building a new index). But
: it turned out that every query using this field for sorting ends in an
: out of memory error. Even sorting result sets containing just one
: document does not work. The field is of type solr.StrField and strange
: enough there are some other fields in the index of the same type which
: do not cause these problems (but not all of them; our uniqueKey-field
: has the same problems with sorting).
: Now I am wondering why sorting works with some of the fields but not
: with others. Could it be that this depends on the content?



-Hoss


Mime
View raw message