lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: Slowly running OOM due to Query instances?!
Date Fri, 07 Jul 2017 13:34:51 GMT
Hello,

There has not been a change in code recently. An increase in QPS and a broader spectrum of
queries (a world wide network of users started using our platform) happened a month ago. All
still happy with 400 MB, nothing changed. The index does increase continously, all the time
so i did expect more BytesRef and stuff to be around, not Query instances, that's just weird.

Thanks,
Markus
 
 
-----Original message-----
> From:Joel Bernstein <joelsolr@gmail.com>
> Sent: Friday 7th July 2017 14:50
> To: solr-user@lucene.apache.org
> Subject: Re: Slowly running OOM due to Query instances?!
> 
> What changed in the system?
> 
> Has there been a code change, increased QPS or different types of queries
> being run?
> 
> 
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Fri, Jul 7, 2017 at 8:07 AM, Markus Jelsma <markus.jelsma@openindex.io>
> wrote:
> 
> > Hello,
> >
> > No, 800 MB heap is fine. Up till now, heap consumption averages at 400 MB
> > for past few months, which is fine. FilterCache and QueryCache are always
> > full. If restarted, they fill up very quickly. FilterCache is just 512
> > entries large, QueryCache about 2k.
> >
> > All Solr instances run on separate machines (VMs).
> >
> > Thanks,
> > Markus
> >
> >
> >
> > -----Original message-----
> > > From:Susheel Kumar <susheel2777@gmail.com>
> > > Sent: Friday 7th July 2017 13:27
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Slowly running OOM due to Query instances?!
> > >
> > > Xms 800m sounds low regardless did you know how much total cache
> > > consumption may go based on your current solrconfig.xml settings. Also 2
> > > shards and 3 replca's are on 6 such machines i assume.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > On Fri, Jul 7, 2017 at 7:01 AM, Markus Jelsma <
> > markus.jelsma@openindex.io>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > This morning i spotted our QTime suddenly go up. This has been going on
> > > > for a few hours by now and coincides with a serious increase in heap
> > > > consumption. No node ran out of memory so far but either that is going
> > to
> > > > happen soon, or the nodes become unusable in another manner.
> > > >
> > > > I restarted one of the Solr instances and launched VisualVM at it, and
> > > > some other nodes that use to much heap. Starting the memory sampler,
> > > > something was obvious straight away.
> > > >
> > > > The nodes consuming too much heap all have a serious amount of *Query,
> > and
> > > > BooleanClause instances, PayloadScoreQuery, TermQuery, BoostQuery,
> > > > BooleanQuery, SpanTermQuery and so forth. Lots of Builder and Term
> > > > instances too, very distinct from the node that was just freshly
> > restarted.
> > > >
> > > > Another peculiarity, some nodes have exactly 65536 instances of
> > TermQuery
> > > > and/or BoostQuery, probably unrelated but not something i would have
> > > > expected to see anyway.
> > > >
> > > > So, what's up? We do have a custom query parser extending
> > EdismaxQParser,
> > > > it transliterates dates and creates payload and span queries. I may be
> > > > doing something wrong but i don't know, i have made and used a variety
> > of
> > > > QParsers, for many years but this is new. Any hints on where to look,
> > what
> > > > to watch out for?
> > > >
> > > > Many thanks!
> > > > Markus
> > > >
> > > > Xmx 800m, 8 GB RAM, SSD
> > > > 2 shards, three replica's
> > > > replica size ~17 GB, 2.2 million docs/replica
> > > >
> > >
> >
> 

Mime
View raw message