lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <wun...@wunderwood.org>
Subject Re: Anyone have any comments on current solr monitoring favorites?
Date Thu, 02 Nov 2017 18:35:52 GMT
We use New Relic for JVM, CPU, and disk monitoring.

I tried the built-in metrics support in 6.4, but it just didn’t do what we want. We want
rates and percentiles for each request handler. That gives us 95th percentile for textbooks
suggest or for homework search results page, etc. The Solr metrics didn’t do that. The Jetty
metrics didn’t do that.

We built a dedicated servlet filter that goes in front of the Solr webapp and reports metrics.
It has some special hacks to handle some weird behavior in SolrJ. A request to the “/srp”
handler is sent as “/select?qt=/srp”, so we normalize that.

The metrics start with the cluster name, the hostname, and the collection. The rest is generated
like this:

URL: GET /solr/textbooks/select?q=foo&qt=/auto
Metric: textbooks.GET./auto

URL: GET /solr/textbooks/select?q=foo
Metric: textbooks.GET./select

URL: GET /solr/questions/auto
Metric: questions.GET./auto

So a full metric for the cluster “solr-cloud” and the host “search01" would look like
“solr-cloud.search01.solr.textbooks.GET./auto.m1_rate”.

We send all that to InfluxDB. We’ve configured a template so that each part of the metric
name is mapped to a field, so we can write efficient queries in InfluxQL.

Metrics are graphed in Grafana. We have dashboards that mix Cloudwatch (for the load balancer)
and InfluxDB.

I’m still working out the kinks in some of the more complicated queries, but the data is
all there. I also want to expand the servlet filter to report HTTP response codes.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 2, 2017, at 9:30 AM, Petersen, Robert (Contr) <robert.petersen1@ftr.com>
wrote:
> 
> OK I'm probably going to open a can of worms here...  lol
> 
> 
> In the old old days I used PSI probe to monitor solr running on tomcat which worked ok
on a machine by machine basis.
> 
> 
> Later I had a grafana dashboard on top of graphite monitoring which was really nice looking
but kind of complicated to set up.
> 
> 
> Even later I successfully just dropped in a newrelic java agent which had solr monitors
and a dashboard right out of the box, but it costs money for the full tamale.
> 
> 
> For basic JVM health and Solr QPS and time percentiles, does anyone have any favorites
or other alternative suggestions?
> 
> 
> Thanks in advance!
> 
> Robi
> 
> ________________________________
> 
> This communication is confidential. Frontier only sends and receives email on the basis
of the terms set out at http://www.frontier.com/email_disclaimer.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message