lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re: Solr large boolean filter
Date Sun, 16 Jun 2013 09:43:13 GMT
Right.
FieldCacheTermsFilter is an option. You need to create own QParserPlugin
which yields FieldCacheTermsFilter, hook him as ..&fq={!idsqp
cache=false}&..
Mind disabling caching! Mind term ecoding due to field type!

I also suggest to check how much it spend for tokenization. Once a day I've
got some profit by using efficient encoding for this param (try fixed
length or vint)

There is a one more gain when the core query is highly selective and id
filter is weakly selective, in  this case using explicit PostFiltering
(what a hack btw) is desired. see
http://yonik.com/posts/advanced-filter-caching-in-solr/

>From my experience the proper solution for such problems is moving to one
of the joins or ExternalFileField.




On Sun, Jun 16, 2013 at 2:49 AM, Igor Kustov <ivkustov@gmail.com> wrote:

> I know i'm not the first one with this problem.
>
> I'm currently using solr 4.2.1 with approximately 10 mln documents in the
> index.
>
> The index is updated frequently.
>
> The filter_query is just a one big boolean or query by id.
>
> fq=id:(1 2 3 4 ... 50950)
>
> ids list is always different and not sequential.
>
> The problem is that query performance not so well, as you can imagine.
>
> In some particular cases i'm able to do filtering based on different
> fields,
> but in some cases (like 30-40% of all queries) i'm still end up with this
> large id filter.
>
> I'm looking for the ways to improve this query performance.
>
> It doesn't seem like solr join could be applied there.
>
> Another option that I found is to somehow use Lucene FieldCacheTermsFilter.
> Does it worth a try?
>
> Maybe i've missed some other options?
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-large-boolean-filter-tp4070747.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mkhludnev@griddynamics.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message