lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn Häuser <bjoernhaeu...@gmail.com>
Subject Re: Heap Memory Problem after Upgrading to 7.4.0
Date Mon, 03 Sep 2018 19:29:40 GMT
Hi Erick,

thank you for your answer.

Unfortunately I do not have a heap dump from 6.6.


> On 3. Sep 2018, at 20:48, Erick Erickson <erickerickson@gmail.com> wrote:
> 
> I would expect at least 1 IndexSearcher per replica, how many total
> replicas hosted in your JVM?

27 replicas per JVM.

> 
> Plus, if you're actively indexing, there may temporarily be 2
> IndexSearchers open while the new searcher warms.
> 
> And there may be quite a few caches, at least queryResultCache and
> filterCache and documentCache, one of each per replica and maybe two
> (for queryResultCache and filterCache) if you have a background
> searcher autowarming.
> 
> At a glance, your autowarm counts are very high, so it may take some
> time to autowarm leading to multiple IndexSearchers and caches open
> per replica when you happen to hit a commit point. I usually start
> with 16-20 as an autowarm count, the benefit decreases rapidly as you
> increase the count.

As a counter measure I reduced the autowarm counts now per API calls to 10. Let me see if
the system is now more stable. Tomorrow morning I will create a new heap dump, to see if the
there are searchers.

Is there any metrics which could tell me that without a heap dump?

> 
> I'm not quite sure why it would be different in 7x .vs. 6x. How much
> heap do you allocate to the JVM? And do you see similar heap dumps in
> 6.6?
> 
> Best,
> Erick

Thanks Erick!

 Björn


> On Mon, Sep 3, 2018 at 10:33 AM Björn Häuser <bjoernhaeuser@gmail.com> wrote:
>> 
>> Hello,
>> 
>> we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard each, 4 replicas
each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We are running Zookeeper 4.1.13.
>> 
>> Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space exhaustion.
After obtaining a heap dump it looks like that we have a lot of IndexSearchers open for our
largest collection.
>> 
>> The dump contains around ~60 IndexSearchers, and each containing around ~40mb heap.
Another 500MB of heap is the fieldcache, which is expected in my opinion.
>> 
>> The current config can be found here: https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844
<https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844>
>> 
>> Analyzing the heap dump eclipse MAT says this:
>> 
>> Problem Suspect 1
>> 
>> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader
@ 0x6807d1048" occupy 1.981.148.336 (38,26%) bytes.
>> 
>> Biggest instances:
>> 
>>        • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - 70.087.272 (1,35%)
bytes.
>>        • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - 65.678.264 (1,27%)
bytes.
>>        • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - 63.050.600 (1,22%)
bytes.
>> 
>> 
>> Problem Suspect 2
>> 
>> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader
@ 0x6807d1048" occupy 1.373.110.208 (26,52%) bytes.
>> 
>> 
>> Any help is appreciated. Thank you very much!
>> Björn


Mime
View raw message