lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: OOM on sorting on dynamic fields
Date Tue, 22 Jun 2010 17:25:18 GMT
Hmmm, I'm missing something here then. Sorting over 15 fields of type long
shouldn't use much memory, even if all the values are unique. When you say
"12-15 dynamic fields", are you talking about 12-15 fields per query out of
XXX total fields? And is XXX large? At a guess, how many different fields
do
you think you're sorting over cumulative by the time you get your OOM?
Note if you sort over the field "erick_time" in 10 different queries, I'm
only counting that as 1 field. I guess another way of asking this is
"how many dynamic fields are there total?".

If this is really a sorting issue, you should be able to force this to
happen
almost immediately by firing off enough sort queries at the server. It'll
tell you a lot if you can't make this happen, even on a relatively small
test machine.

Best
Erick

On Tue, Jun 22, 2010 at 12:59 PM, Matteo Fiandesio <
matteo.fiandesio@gmail.com> wrote:

> Hi Erick,
> the index is quite small (1691145 docs) but sorting is massive and
> often on unique timestamp fields.
>
> OOM occur after a range of time between three and four hours.
> Depending as well if users browse a part of the application.
>
> We use solrj to make the queries so we did not use Readers objects
> directly.
>
> Without sorting we don't see the problem
> Regards,
> Matteo
>
> On 22 June 2010 17:01, Erick Erickson <erickerickson@gmail.com> wrote:
> > Hmmmm.. A couple of details I'm wondering about. How many
> > documents are we talking about in your index? Do you get
> > OOMs when you start fresh or does it take a while?
> >
> > You've done some good investigations, so it seems like there
> > could well be something else going on here than just "the usual
> > suspects" of sorting....
> >
> > I'm wondering if you aren't really closing readers somehow.
> > Are you updating your index frequently and re-opening readers often?
> > If so, how?
> >
> > I'm assuming that if you do NOT sort on all these fields, you don't have
> > the problem, is that true?
> >
> > Best
> > Erick
> >
> > On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
> > matteo.fiandesio@gmail.com> wrote:
> >
> >> Hello,
> >> we are experiencing OOM exceptions in our single core solr instance
> >> (on a (huge) amazon EC2 machine).
> >> We investigated a lot in the mailing list and through jmap/jhat dump
> >> analyzing and the problem resides in the lucene FieldCache that fills
> >> the heap and blows up the server.
> >>
> >> Our index is quite small but we have a lot of sort queries  on fields
> >> that are dynamic,of type long representing timestamps and are not
> >> present in all the documents.
> >> Those queries apply sorting on 12-15 of those fields.
> >>
> >> We are using solr 1.4 in production and the dump shows a lot of
> >> Integer/Character and Byte Array filled up with 0s.
> >> With solr's trunk code things does not change.
> >>
> >> In the mailing list we saw a lot of messages related to this issues:
> >> we tried truncating the dates to day precision,using missingSortLast =
> >> true,changing the field type from slong to long,setting autowarming to
> >> different values,disabling and enabling caches with different values
> >> but we did not manage to solve the problem.
> >>
> >> We were thinking to implement an LRUFieldCache field type to manage
> >> the FieldCache as an LRU and preventing but, before starting a new
> >> development, we want to be sure that we are not doing anything wrong
> >> in the solr configuration or in the index generation.
> >>
> >> Any help would be appreciated.
> >> Regards,
> >> Matteo
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message