lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davis, Daniel (NIH/NLM) [C]" <daniel.da...@nih.gov>
Subject RE: Measuring QPS
Date Mon, 06 Apr 2015 18:24:41 GMT
Its very common to do autocomplete based on popular queries/titles over some sliding time window.
  Some enterprise search systems even apply age weighting so that they don't need to re-index
but continuously add to the index.   This way, they can do autocomplete based on what's popular
these days.

We use relevance/field boosts/phrase matching etc. to get the best guess about what results
they want to see.   This is similar - we use relevance, field boosting to guess what users
want to search for.   Zipf's law applies to searches as well as results.

-----Original Message-----
From: Siegfried Goeschl [mailto:sgoeschl@gmx.at] 
Sent: Monday, April 06, 2015 2:17 PM
To: solr-user@lucene.apache.org
Subject: Re: Measuring QPS

Hi Daniel,

interesting - I never thought of autocompletion but for keeping track of user behaviour :-)

* the numbers are helpful for the online advertisement team to sell campaigns
* it is used for sanity checks - sensible queries returning no results or returning too many
results

Cheers,

Siegfried Goeschl

> On 06 Apr 2015, at 20:04, Davis, Daniel (NIH/NLM) [C] <daniel.davis@nih.gov> wrote:
> 
> Siegfried,
> 
> It is early days as yet.   I don't think we need a code drop.   AFAIK, none of our current
Solr applications autocomplete the search box based on popular query/title keywords.   We
have other applications that do that, but they don't use Solr.
> 
> Thanks again,
> 
> Dan
> 
> -----Original Message-----
> From: Siegfried Goeschl [mailto:sgoeschl@gmx.at]
> Sent: Monday, April 06, 2015 1:42 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Measuring QPS
> 
> Hi Dan,
> 
> at willhaben.at (customer of mine) two SOLR components were written 
> for SOLR 3 and ported to SORL 4
> 
> 1) SlowQueryLog which dumps long-running search requests into a log 
> file
> 
> 2) Most Frequent Search Terms allowing to query & filter the most 
> frequent user search terms over the browser
> 
> Some notes along the line
> 
> 
> * For both components I have the “GO" to open source them but I never 
> had enough time to do that (shame on me) - see 
> https://issues.apache.org/jira/browse/SOLR-4056
> 
> * The Most Frequent Search Term component actually mimics a SOLR 
> server you feed the user search terms so this might be a better 
> solution in the long run. But this requires to have a separate SOLR 
> core & ingest  plus GUI (check out SILK or ELK) - in other words more 
> moving parts in production :-)
> 
> * If there is sufficient interest I can make a code drop on GitHub
> 
> Cheers,
> 
> Siegfried Goeschl
> 
> 
> 
>> On 06 Apr 2015, at 16:25, Davis, Daniel (NIH/NLM) [C] <daniel.davis@nih.gov>
wrote:
>> 
>> Siegfried,
>> 
>> This is a wonderful find.   The second presentation is a nice write-up of a large
number of free tools.   The first presentation prompts a question - did you add custom request
handlers/code to automate determination of best user search terms?   Did any of your custom
work end-up in Solr?
>> 
>> Thank you so much,
>> 
>> Dan
>> 
>> P.S. - your first presentation takes me back to seeing "Angrif der Klonkrieger" in
Berlin after a conference - Hayden Christensen was less annoying in German, because my wife
and I don't speak German ;)   I haven't thought of that in a while.
>> 
>> -----Original Message-----
>> From: Siegfried Goeschl [mailto:sgoeschl@gmx.at]
>> Sent: Saturday, April 04, 2015 4:54 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Measuring QPS
>> 
>> Hi Dan,
>> 
>> I’m using JavaMelody for my SOLR production servers - gives you the 
>> relevant HTTP stats (what’s happening now & historical data) plus JVM 
>> monitoring as additional benefit. The servers are deployed on Tomcat 
>> so I’m of little help regarding Jetty - having said that
>> 
>> * you need two Jars (javamelody & robin)
>> * tinker with web.xml
>> 
>> Here are two of my presentations mentioning JavaMelody (plus some 
>> other stuff)
>> 
>> http://people.apache.org/~sgoeschl/presentations/solr-from-developmen
>> t
>> -to-production-20121210.pdf
>> <http://people.apache.org/~sgoeschl/presentations/solr-from-developme
>> n
>> t-to-production-20121210.pdf>
>> http://people.apache.org/~sgoeschl/presentations/jsug-2015/jee-perfor
>> m
>> ance-monitoring.pdf
>> <http://people.apache.org/~sgoeschl/presentations/jsug-2015/jee-perfo
>> r
>> mance-monitoring.pdf>
>> 
>> Cheers,
>> 
>> Siegfried Goeschl
>> 
>>> On 03 Apr 2015, at 17:53, Shawn Heisey <apache@elyograg.org> wrote:
>>> 
>>> On 4/3/2015 9:37 AM, Davis, Daniel (NIH/NLM) [C] wrote:
>>>> I wanted to gather QPS for our production Solr instances, but I was surprised
that the Admin UI did not contain this information.   We are running a mix of versions, but
mostly 4.10 at this point.   We are not using SolrCloud at present; that's part of why I'm
checking - I want to validate the size of our existing setup and what sort of SolrCloud setup
would be needed to centralize several of them.
>>>> 
>>>> What is the best way to gather QPS information?
>>>> 
>>>> What is the best way to add information like this to the Admin UI, if I decide
to take that step?
>>> 
>>> As of Solr 4.1 (three years ago), request rate information is 
>>> available in the admin UI and via JMX.  In the admin UI, choose a 
>>> core from the dropdown, click on Plugins/Stats, then QUERYHANDLER, 
>>> and open the handler you wish to examine.  You have 
>>> avgRequestsPerSecond, which is calculated for the entire runtime of 
>>> the SolrCore, as well as 5minRateReqsPerSecond and 
>>> 15minRateReqsPerSecond, which are far more useful pieces of information.
>>> 
>>> https://issues.apache.org/jira/browse/SOLR-1972
>>> 
>>> Thanks,
>>> Shawn
>>> 
>> 
> 

Mime
View raw message