lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Heap Memory Problem after Upgrading to 7.4.0
Date Mon, 03 Sep 2018 18:48:48 GMT
I would expect at least 1 IndexSearcher per replica, how many total
replicas hosted in your JVM?

Plus, if you're actively indexing, there may temporarily be 2
IndexSearchers open while the new searcher warms.

And there may be quite a few caches, at least queryResultCache and
filterCache and documentCache, one of each per replica and maybe two
(for queryResultCache and filterCache) if you have a background
searcher autowarming.

At a glance, your autowarm counts are very high, so it may take some
time to autowarm leading to multiple IndexSearchers and caches open
per replica when you happen to hit a commit point. I usually start
with 16-20 as an autowarm count, the benefit decreases rapidly as you
increase the count.

I'm not quite sure why it would be different in 7x .vs. 6x. How much
heap do you allocate to the JVM? And do you see similar heap dumps in
6.6?

Best,
Erick
On Mon, Sep 3, 2018 at 10:33 AM Björn Häuser <bjoernhaeuser@gmail.com> wrote:
>
> Hello,
>
> we recently upgraded our solrcloud (5 nodes, 25 collections, 1 shard each, 4 replicas
each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We are running Zookeeper 4.1.13.
>
> Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap space exhaustion. After
obtaining a heap dump it looks like that we have a lot of IndexSearchers open for our largest
collection.
>
> The dump contains around ~60 IndexSearchers, and each containing around ~40mb heap. Another
500MB of heap is the fieldcache, which is expected in my opinion.
>
> The current config can be found here: https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844
<https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844>
>
> Analyzing the heap dump eclipse MAT says this:
>
> Problem Suspect 1
>
> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader
@ 0x6807d1048" occupy 1.981.148.336 (38,26%) bytes.
>
> Biggest instances:
>
>         • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - 70.087.272 (1,35%)
bytes.
>         • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - 65.678.264 (1,27%)
bytes.
>         • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - 63.050.600 (1,22%)
bytes.
>
>
> Problem Suspect 2
>
> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader
@ 0x6807d1048" occupy 1.373.110.208 (26,52%) bytes.
>
>
> Any help is appreciated. Thank you very much!
> Björn

Mime
View raw message