lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Out of memory during the indexing
Date Mon, 05 Dec 2011 13:38:20 GMT
There's no good way to say to Solr "Use only this
much memory for searching". You can certainly
limit the size somewhat by configuring your caches
to be small. But if you're sorting, then Lucene will
use up some cache space etc.

Are you actually running into problems?

Best
Erick

On Fri, Dec 2, 2011 at 2:26 PM, Jeff Crump <jeffrey.crump@gmail.com> wrote:
> Can anyone advise techniques for limiting the size of the RAM buffers to
> begin with?  As the index grows, I shouldn't have to keep increasing the
> heap.  We have a high-ingest, low-query-rate environment and I'm not as
> much concerned with the query-time caches as I am with the segment core
> readers/SolrIndexSearchers themselves.
>
> On 9 November 2011 06:10, Andre Bois-Crettez <andre.bois@kelkoo.com> wrote:
>
>> How much memory you actually allocate to the JVM ?
>> http://wiki.apache.org/solr/**SolrPerformanceFactors#Memory_**
>> allocated_to_the_Java_VM<http://wiki.apache.org/solr/SolrPerformanceFactors#Memory_allocated_to_the_Java_VM>
>> You need to increase the -Xmx value, otherwise your large ram buffers
>> won't fit in the java heap.
>>
>>
>>
>> sivaprasad wrote:
>>
>>> Hi,
>>>
>>> I am getting the following error during the indexing.I am trying to index
>>> 14
>>> million records but the document size is very minimal.
>>>
>>> *Error:*
>>> 2011-11-08 14:53:24,634 ERROR [STDERR] (Thread-12)
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>
>>>
>>>
>> [...]
>>
>>  Do i need to increase the heap size for JVM?
>>>
>>> My solrconfig settings are given below.
>>>
>>> <indexDefaults>
>>>      <useCompoundFile>false</**useCompoundFile>
>>>
>>>    <mergeFactor>25</mergeFactor>
>>>        <maxBufferedDocs>2</**maxBufferedDocs>
>>>       <ramBufferSizeMB>1024</**ramBufferSizeMB>
>>>    <maxMergeDocs>2147483647</maxMergeDocs>
>>>    <maxFieldLength>10000</**maxFieldLength>
>>>    <writeLockTimeout>1000</**writeLockTimeout>
>>>    <commitLockTimeout>10000</**commitLockTimeout>
>>>
>>> and the main index values are
>>> <useCompoundFile>false</**useCompoundFile>
>>>    <ramBufferSizeMB>512</**ramBufferSizeMB>
>>>    <mergeFactor>10</mergeFactor>
>>>    <maxMergeDocs>2147483647</maxMergeDocs>
>>>    <maxFieldLength>10000</**maxFieldLength>
>>>
>>> Do i need to increase the ramBufferSizeMB to a little higher?
>>>
>>> Please provide your inputs.
>>>
>>> Regards,
>>> Siva
>>>
>>> --
>>> View this message in context: http://lucene.472066.n3.**
>>> nabble.com/Out-of-memory-**during-the-indexing-**tp3492701p3492701.html<http://lucene.472066.n3.nabble.com/Out-of-memory-during-the-indexing-tp3492701p3492701.html>
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>> --
>> André Bois-Crettez
>>
>> Search technology, Kelkoo
>> http://www.kelkoo.com/
>>
>>

Mime
View raw message