lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: OOM on sorting on dynamic fields
Date Tue, 22 Jun 2010 15:01:47 GMT
Hmmmm.. A couple of details I'm wondering about. How many
documents are we talking about in your index? Do you get
OOMs when you start fresh or does it take a while?

You've done some good investigations, so it seems like there
could well be something else going on here than just "the usual
suspects" of sorting....

I'm wondering if you aren't really closing readers somehow.
Are you updating your index frequently and re-opening readers often?
If so, how?

I'm assuming that if you do NOT sort on all these fields, you don't have
the problem, is that true?

Best
Erick

On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
matteo.fiandesio@gmail.com> wrote:

> Hello,
> we are experiencing OOM exceptions in our single core solr instance
> (on a (huge) amazon EC2 machine).
> We investigated a lot in the mailing list and through jmap/jhat dump
> analyzing and the problem resides in the lucene FieldCache that fills
> the heap and blows up the server.
>
> Our index is quite small but we have a lot of sort queries  on fields
> that are dynamic,of type long representing timestamps and are not
> present in all the documents.
> Those queries apply sorting on 12-15 of those fields.
>
> We are using solr 1.4 in production and the dump shows a lot of
> Integer/Character and Byte Array filled up with 0s.
> With solr's trunk code things does not change.
>
> In the mailing list we saw a lot of messages related to this issues:
> we tried truncating the dates to day precision,using missingSortLast =
> true,changing the field type from slong to long,setting autowarming to
> different values,disabling and enabling caches with different values
> but we did not manage to solve the problem.
>
> We were thinking to implement an LRUFieldCache field type to manage
> the FieldCache as an LRU and preventing but, before starting a new
> development, we want to be sure that we are not doing anything wrong
> in the solr configuration or in the index generation.
>
> Any help would be appreciated.
> Regards,
> Matteo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message