lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Harris" <rygu...@gmail.com>
Subject Re: Big slowdown with phrase queries
Date Sun, 13 Jul 2008 02:00:33 GMT
Just for fun, I ran my cpu profiler against Solr while running a
little app to hammer Solr with phrase queries. As Mike and Yonik could
have predicted, I found that Solr was spending basically all its time
in lucene.index.MultiSegmentReader$MultiTermPositions code (called
from lucene.search.PhrasePositions code), which is to say practically
all it was doing was phrase position I/O.

Mike, your idea of indexing bigrams is also interesting. Do you know
if any text search platforms do this behind the scenes as their
default way of handling phrase queries? See, I have a copy of
dtSearch, and when I use it to index the same data I've been
discussing here, on the same machine (and end with roughly the same
size index, measured in GB), it performs notably faster than Solr on
phrase queries. I haven't done any serious tests, but it might be an
order of magnitude difference or more. Now dtSearch may have a speed
advantage over Solr because it's written in C and it's optimized for
Windows in particular, but I'm wondering if the real explanation
wouldn't be a difference in algorithms. (Hopefully the explanation is
not me being stupid and thinking I've indexed the same thing in
dtSearch when in fact I've configured them with some wildly different
settings.) Is this plausible?

On Thu, Jul 3, 2008 at 5:30 PM, Mike Klaas <mike.klaas@gmail.com> wrote:
>
> On 3-Jul-08, at 5:13 PM, Chris Harris wrote:
>
>>> That's pretty much impossible (way too small).  Double check those
>>> numbers.
>>
>> I don't know where I got the above numbers. Sorry. Here are the real
>> numbers:
>>
>> .tis file: 730MB
>> .frq files: 10.1 GB
>> .prx file: 43.2 GB
>>
>> Now keeping all *that* in RAM, that sounds like a challenge.
>
> It doesn't have to be *all* in RAM... the OS will figure out what parts are
> needed.
>
> One alternative you might consider is using a flash hard drive.  Another is
> to index bigrams as terms, and do phrase queries using the conjunction of
> the bigrams of a phrase.  This should make phrase queries only a few times
> slower than term queries, and probably inflate your .frq to "only" 25GB
> (.prx could be ignored).
>
> Some other tricks, like stop word removal, also speed up phrase queries.
>
> -Mike
>

Mime
View raw message