lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <wunderw...@netflix.com>
Subject Re: Big slowdown with phrase queries
Date Sun, 13 Jul 2008 02:31:43 GMT
On 7/12/08 7:00 PM, "Chris Harris" <ryguasu@gmail.com> wrote:
>
> Mike, your idea of indexing bigrams is also interesting. Do you know
> if any text search platforms do this behind the scenes as their
> default way of handling phrase queries?

Infoseek indexed biwords with their Ultra engine, which lives
on as the Ultraseek enterprise engine. This makes phrase
search very fast, but it isn't exact.

A few years ago, Doug Cutting told me they were doing something a
bit more clever with Nutch, using a biword search to make a candidate
set of phrase matches, then using positional (exact) phrase search
against just that set to get the true matches. That is a really nice
trick that combines the speed of biword matches with the exactness
of positional search.

Biwords are also useful for an approximate phrase IDF. Use the
least common biword and say that is the IDF for the phrase.

I believe Infoseek got the idea through William Chang, who had
been doing genomic search before that. William is at Baidu now.

Infoseek Ultra launched summer of 1996.

wunder




Mime
View raw message