lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matteo Fiandesio <matteo.fiande...@gmail.com>
Subject Re: OOM on sorting on dynamic fields
Date Mon, 21 Jun 2010 09:08:47 GMT
Compiling solr with lucene 2.9.3 instead of 2.9.1 will solve this issue?
Regards,
Matteo

On 19 June 2010 02:28, Lance Norskog <goksron@gmail.com> wrote:
> The Lucene implementation of sorting creates an array of four-byte
> ints for every document in the index, and another array of the unique
> values in the field.
> If the timestamps are 'date' or 'tdate' in the schema, they do not
> need the second array.
>
> You can also sort by a field's with a function query. This does not
> build the arrays, but might be a little slower.
> Yes, the sort arrays (and also facet values for a field) should be
> controlled by a fixed-size cache, but they are not.
>
> On Fri, Jun 18, 2010 at 7:52 AM, Matteo Fiandesio
> <matteo.fiandesio@gmail.com> wrote:
>> Hello,
>> we are experiencing OOM exceptions in our single core solr instance
>> (on a (huge) amazon EC2 machine).
>> We investigated a lot in the mailing list and through jmap/jhat dump
>> analyzing and the problem resides in the lucene FieldCache that fills
>> the heap and blows up the server.
>>
>> Our index is quite small but we have a lot of sort queries  on fields
>> that are dynamic,of type long representing timestamps and are not
>> present in all the documents.
>> Those queries apply sorting on 12-15 of those fields.
>>
>> We are using solr 1.4 in production and the dump shows a lot of
>> Integer/Character and Byte Array filled up with 0s.
>> With solr's trunk code things does not change.
>>
>> In the mailing list we saw a lot of messages related to this issues:
>> we tried truncating the dates to day precision,using missingSortLast =
>> true,changing the field type from slong to long,setting autowarming to
>> different values,disabling and enabling caches with different values
>> but we did not manage to solve the problem.
>>
>> We were thinking to implement an LRUFieldCache field type to manage
>> the FieldCache as an LRU and preventing but, before starting a new
>> development, we want to be sure that we are not doing anything wrong
>> in the solr configuration or in the index generation.
>>
>> Any help would be appreciated.
>> Regards,
>> Matteo
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>

Mime
View raw message