lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Benedetti <benedetti.ale...@gmail.com>
Subject Re: Filter queries & caching
Date Tue, 20 Dec 2016 15:04:16 GMT
Hi Erick,
just thinking on this :

1) "q=myclause AND filter(field:value)

is identical to

2) q=myclause&fq=field:value"

Correct me if I am wrong,
those two queries , from filter caching point of view are identical, but
from scoring point of you :

1) will score only the documents resulting from the intersection

2) will score all the documents in myClause and then intersect with the not
ordered docIds list coming from the filter query.

Am I right ? I didn't take a look into the code, so I am just guessing,
Cheers

On Tue, May 10, 2016 at 4:30 PM, Erick Erickson <erickerickson@gmail.com>
wrote:

> No. Please re-read and use the admin plugins/stats page to examine for
> yourself.
>
> 1)  fq=filter(fromfield:[* TO NOW/DAY+1DAY]&& tofield:[NOW/DAY-7DAY TO *])
> && fq=type:abc
>
> && is totally unnecessary when using fq clauses, there is already an
> implicit AND.
> I'm not even sure what the above does, I don't quite know off the top of
> my head
> how that would be parsed.
>
> fq=filter() is unnecessary and in fact (apparently) uses extra
> filterCache entries
> to no purpose.
>
> I'm guessing you're thinking of something like this
>
> q=*:*&fq=(fromfield:[* TO NOW/DAY+1DAY] && tofield:[NOW/DAY-7DAY TO
> *])&fq=type:abc
>
> Would use 2 filterCache entries,
>
> or maybe this: (notice this is "q=" not "fq=")
>
> q=filter(fromfield:[* TO NOW/DAY+1DAY] && tofield:[NOW/DAY-7DAY TO *])
> && filter(type:abc)
>
> would use two filter queries as well. Same thing essentially.
>
> 2) fq= fromfield:[* TO NOW/DAY+1DAY]&& fq=tofield:[NOW/DAY-7DAY TO *]) &&
> fq=type:abc
>
> This is syntactically incorrect, I assume you meant (added left paren
> and again the && is unnecessary):
>
> q=*:*&fq=(fromfield:[* TO NOW/DAY+1DAY] && fq=tofield:[NOW/DAY-7DAY TO *])&
> fq=type:abc
>
> As above the rewritten form would use two filterCache entries.
>
> Best,
> Erick
>
> On Mon, May 9, 2016 at 11:03 PM, Jay Potharaju <jspotharaju@gmail.com>
> wrote:
> > Thanks for the explanation Eric.
> >
> > So that I understand this clearly....
> >
> >
> > 1)  fq=filter(fromfield:[* TO NOW/DAY+1DAY]&& tofield:[NOW/DAY-7DAY TO
> *])
> > && fq=type:abc
> > 2) fq= fromfield:[* TO NOW/DAY+1DAY]&& fq=tofield:[NOW/DAY-7DAY TO *]) &&
> > fq=type:abc
> >
> > Using 1) would benefit from having 2 separate filter caches instead of 3
> > slots in the cache. But in general both would be using the filter cache.
> > And secondly it would  be more useful to use filter() in a scenario like
> > above(mentioned in your email).
> > Thanks
> >
> >
> >
> >
> > On Mon, May 9, 2016 at 9:43 PM, Erick Erickson <erickerickson@gmail.com>
> > wrote:
> >
> >> You're confusing a query clause with fq when thinking about filter() I
> >> think.
> >>
> >> Essentially they don't need to be used together, i.e.
> >>
> >> q=myclause AND filter(field:value)
> >>
> >> is identical to
> >>
> >> q=myclause&fq=field:value
> >>
> >> both in docs returned and filterCache usage.
> >>
> >> q=myclause&filter(fq=field:value)
> >>
> >> actually uses two filterCache entries, so is probably not what you want
> to
> >> use.
> >>
> >> the filter() syntax attached to a q clause (not an fq clause) is meant
> >> to allow you to get speedups
> >> you want to use compound clauses without having every combination be
> >> separate filterCache entries.
> >>
> >> Consider the following:
> >> fq=A OR B
> >> fq=A AND B
> >> fq=A
> >> fq=B
> >>
> >> These would require 4 filterCache entries.
> >>
> >> q=filter(A) OR filter(B)
> >> q=filter(A) AND filter(B)
> >> q=filter(A)
> >> q=filter(B)
> >>
> >> would only require two. Yet all of them would be satisfied only by
> >> looking at the filterCache.
> >>
> >> Aside from the example immediately above, which one you use is largely
> >> a matter of taste.
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, May 9, 2016 at 12:47 PM, Jay Potharaju <jspotharaju@gmail.com>
> >> wrote:
> >> > Thanks Ahmet...but I am not still clear how is adding filter() option
> >> > better or is it the same as filtercache?
> >> >
> >> > My question is below.
> >> >
> >> > "As mentioned above adding filter() will add the filter query to the
> >> cache.
> >> > This would mean that results are fetched from cache instead of
> running n
> >> > number of filter queries  in parallel.
> >> > Is it necessary to use the filter() option? I was under the impression
> >> that
> >> > all filter queries will get added to the "filtercache". What is the
> >> > advantage of using filter()?"
> >> >
> >> > Thanks
> >> >
> >> > On Sun, May 8, 2016 at 6:30 PM, Ahmet Arslan
> <iorixxx@yahoo.com.invalid>
> >> > wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> As I understand it useful incase you use an OR operator between two
> >> >> restricting clauses.
> >> >> Recall that multiple fq means implicit AND.
> >> >>
> >> >> ahmet
> >> >>
> >> >>
> >> >>
> >> >> On Monday, May 9, 2016 4:02 AM, Jay Potharaju <jspotharaju@gmail.com
> >
> >> >> wrote:
> >> >> As mentioned above adding filter() will add the filter query to the
> >> cache.
> >> >> This would mean that results are fetched from cache instead of
> running n
> >> >> number of filter queries  in parallel.
> >> >> Is it necessary to use the filter() option? I was under the
> impression
> >> that
> >> >> all filter queries will get added to the "filtercache". What is the
> >> >> advantage of using filter()?
> >> >>
> >> >> *From
> >> >> doc:
> >> >>
> >> https://cwiki.apache.org/confluence/display/solr/Query+
> Settings+in+SolrConfig
> >> >> <
> >> >>
> >> https://cwiki.apache.org/confluence/display/solr/Query+
> Settings+in+SolrConfig
> >> >> >*
> >> >> This cache is used by SolrIndexSearcher for filters (DocSets) for
> >> unordered
> >> >> sets of all documents that match a query. The numeric attributes
> control
> >> >> the number of entries in the cache.
> >> >> Solr uses the filterCache to cache results of queries that use the
fq
> >> >> search parameter. Subsequent queries using the same parameter setting
> >> >> result in cache hits and rapid returns of results. See Searching for
> a
> >> >> detailed discussion of the fq parameter.
> >> >>
> >> >> *From Yonik's site: http://yonik.com/solr/query-syntax/#FilterQuery
> >> >> <http://yonik.com/solr/query-syntax/#FilterQuery>*
> >> >>
> >> >> (Since Solr 5.4)
> >> >>
> >> >> A filter query retrieves a set of documents matching a query from the
> >> >> filter cache. Since scores are not cached, all documents that match
> the
> >> >> filter produce the same score (0 by default). Cached filters will be
> >> >> extremely fast when they are used again in another query.
> >> >>
> >> >>
> >> >> Thanks
> >> >>
> >> >>
> >> >> On Fri, May 6, 2016 at 9:46 AM, Jay Potharaju <jspotharaju@gmail.com
> >
> >> >> wrote:
> >> >>
> >> >> > We have high query load and considering that I think the
> suggestions
> >> made
> >> >> > above will help with performance.
> >> >> > Thanks
> >> >> > Jay
> >> >> >
> >> >> > On Fri, May 6, 2016 at 7:26 AM, Shawn Heisey <apache@elyograg.org>
> >> >> wrote:
> >> >> >
> >> >> >> On 5/6/2016 7:19 AM, Shawn Heisey wrote:
> >> >> >> > With three separate
> >> >> >> > fq parameters, you'll get three cache entries in filterCache
> from
> >> the
> >> >> >> > one query.
> >> >> >>
> >> >> >> One more tidbit of information related to this:
> >> >> >>
> >> >> >> When you have multiple filters and they aren't cached, I am
> >> reasonably
> >> >> >> certain that they run in parallel.  Instead of one complex
filter,
> >> you
> >> >> >> would have three simple filters running simultaneously.  For
low
> to
> >> >> >> medium query loads on a server with a whole bunch of CPUs,
where
> >> there
> >> >> >> is plenty of spare CPU power, this can be a real gain in
> performance
> >> ...
> >> >> >> but if the query load is really high, it might be a bad thing.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Shawn
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Thanks
> >> >> > Jay Potharaju
> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thanks
> >> >> Jay Potharaju
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks
> >> > Jay Potharaju
> >>
> >
> >
> >
> > --
> > Thanks
> > Jay Potharaju
>



-- 
--------------------------

Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message