lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: Heap Memory Problem after Upgrading to 7.4.0
Date Thu, 06 Sep 2018 08:29:20 GMT
Thanks Tomás!

Björn, can you reproduce the problem in a local and controlled environment?

Markus

 
 
-----Original message-----
> From:Tomás Fernández Löbbe <tomasflobbe@gmail.com>
> Sent: Wednesday 5th September 2018 18:32
> To: solr-user@lucene.apache.org
> Subject: Re: Heap Memory Problem after Upgrading to 7.4.0
> 
> I think this is pretty bad. I created
> https://issues.apache.org/jira/browse/SOLR-12743. Feel free to add any more
> details you have there.
> 
> On Mon, Sep 3, 2018 at 1:50 PM Markus Jelsma <markus.jelsma@openindex.io>
> wrote:
> 
> > Hello Björn,
> >
> > Take great care, 7.2.1 cannot read an index written by 7.4.0, so you
> > cannot roll back but need to reindex!
> >
> > Andrey Kudryavtsev made a good suggestion in the thread on how to find the
> > culprit, but it will be a tedious task. I have not yet had the time or
> > courage to venture there.
> >
> > Hope it helps,
> > Markus
> >
> >
> >
> > -----Original message-----
> > > From:Björn Häuser <bjoernhaeuser@gmail.com>
> > > Sent: Monday 3rd September 2018 22:28
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Heap Memory Problem after Upgrading to 7.4.0
> > >
> > > Hi Markus,
> > >
> > > this reads exactly like what we have. Where you able to figure out
> > anything? Currently thinking about rollbacking to 7.2.1.
> > >
> > >
> > >
> > > > On 3. Sep 2018, at 21:54, Markus Jelsma <markus.jelsma@openindex.io>
> > wrote:
> > > >
> > > > Hello,
> > > >
> > > > Getting an OOM plus the fact you are having a lot of IndexSearcher
> > instances rings a familiar bell. One of our collections has the same issue
> > [1] when we attempted an upgrade 7.2.1 > 7.3.0. I managed to rule out all
> > our custom Solr code but had to keep our Lucene filters in the schema, the
> > problem persisted.
> > > >
> > > > The odd thing, however, is that you appear to have the same problem,
> > but not with 7.3.0? Since you shortly after 7.3.0 upgraded to 7.4.0, can
> > you confirm the problem is not also in 7.3.0?
> > > >
> > >
> > > We had very similar problems with 7.3.0 but never analyzed them and just
> > updated to 7.4.0 because I thought thats the bug we hit:
> > https://issues.apache.org/jira/browse/SOLR-11882 <
> > https://issues.apache.org/jira/browse/SOLR-11882>
> > >
> > >
> > > > You should see the instance count for IndexSearcher increase by one
> > for each replica on each commit.
> > >
> > >
> > > Sorry, where can I find this? ;) Sorry, did not find anything.
> > >
> > > Thanks
> > > Björn
> > >
> > > >
> > > > Regards,
> > > > Markus
> > > >
> > > > [1]
> > http://lucene.472066.n3.nabble.com/RE-7-3-appears-to-leak-td4396232.html
> > > >
> > > >
> > > >
> > > > -----Original message-----
> > > >> From:Erick Erickson <erickerickson@gmail.com>
> > > >> Sent: Monday 3rd September 2018 20:49
> > > >> To: solr-user <solr-user@lucene.apache.org>
> > > >> Subject: Re: Heap Memory Problem after Upgrading to 7.4.0
> > > >>
> > > >> I would expect at least 1 IndexSearcher per replica, how many total
> > > >> replicas hosted in your JVM?
> > > >>
> > > >> Plus, if you're actively indexing, there may temporarily be 2
> > > >> IndexSearchers open while the new searcher warms.
> > > >>
> > > >> And there may be quite a few caches, at least queryResultCache and
> > > >> filterCache and documentCache, one of each per replica and maybe two
> > > >> (for queryResultCache and filterCache) if you have a background
> > > >> searcher autowarming.
> > > >>
> > > >> At a glance, your autowarm counts are very high, so it may take some
> > > >> time to autowarm leading to multiple IndexSearchers and caches open
> > > >> per replica when you happen to hit a commit point. I usually start
> > > >> with 16-20 as an autowarm count, the benefit decreases rapidly as
you
> > > >> increase the count.
> > > >>
> > > >> I'm not quite sure why it would be different in 7x .vs. 6x. How much
> > > >> heap do you allocate to the JVM? And do you see similar heap dumps
in
> > > >> 6.6?
> > > >>
> > > >> Best,
> > > >> Erick
> > > >> On Mon, Sep 3, 2018 at 10:33 AM Björn Häuser <bjoernhaeuser@gmail.com>
> > wrote:
> > > >>>
> > > >>> Hello,
> > > >>>
> > > >>> we recently upgraded our solrcloud (5 nodes, 25 collections, 1
shard
> > each, 4 replicas each) from 6.6.0 to 7.3.0 and shortly after to 7.4.0. We
> > are running Zookeeper 4.1.13.
> > > >>>
> > > >>> Since the upgrade to 7.3.0 and also 7.4.0 we encountering heap
space
> > exhaustion. After obtaining a heap dump it looks like that we have a lot of
> > IndexSearchers open for our largest collection.
> > > >>>
> > > >>> The dump contains around ~60 IndexSearchers, and each containing
> > around ~40mb heap. Another 500MB of heap is the fieldcache, which is
> > expected in my opinion.
> > > >>>
> > > >>> The current config can be found here:
> > https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844 <
> > https://gist.github.com/bjoernhaeuser/327a65291ac9793e744b87f0a561e844>
> > > >>>
> > > >>> Analyzing the heap dump eclipse MAT says this:
> > > >>>
> > > >>> Problem Suspect 1
> > > >>>
> > > >>> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded
> > by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy
> > 1.981.148.336 (38,26%) bytes.
> > > >>>
> > > >>> Biggest instances:
> > > >>>
> > > >>>        • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8
-
> > 70.087.272 (1,35%) bytes.
> > > >>>        • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040
-
> > 65.678.264 (1,27%) bytes.
> > > >>>        • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680
-
> > 63.050.600 (1,22%) bytes.
> > > >>>
> > > >>>
> > > >>> Problem Suspect 2
> > > >>>
> > > >>> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded
> > by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy
> > 1.373.110.208 (26,52%) bytes.
> > > >>>
> > > >>>
> > > >>> Any help is appreciated. Thank you very much!
> > > >>> Björn
> > > >>
> > >
> > >
> >
> 

Mime
View raw message