nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki>
Subject Re: IndexOptimizer (Re: Lucene performance bottlenecks)
Date Thu, 15 Dec 2005 09:53:44 GMT
Doug Cutting wrote:

> Andrzej Bialecki wrote:
>> I tested it on a 5 mln index.
> Thanks, this is great data!
> Can you please tell a bit more about the experiments?  In particular:
>  . How were scores assigned to pages?  Link analysis?  log(number of 
> incoming links) or OPIC?


>  . How were the queries generated?  From a log or randomly?

Queries have been picked up manually, to test the worst performing cases 
from a real query log.

>  . How many queries did you test with?

A couple, not many. This of course needs to be improved, but I didn't 
want to spend too much time until we polish the code.

>  . When results differed greatly, did they look a lot worse?

Yes. E.g. see the differences for MAX_HITS=10000

> My attempt to sort a 38M page index failed with OutOfMemory.  Sigh.
>> For MAX_HITS=1000 the performance increase was ca. 40-fold, i.e. 
>> queries, which executed in e.g. 500 ms now executed in 10-20ms 
>> (perfRate=40). Following the intuition, performance drops as we 
>> increase MAX_HITS, until it reaches a more or less original values 
>> (perfRate=1) for MAX_HITS=300000 (for a 5 mln doc index). After that, 
>> increasing MAX_HITS actually worsens the performance (perfRate << 1) 
>> - which can be explained by the fact that the standard HitCollector 
>> doesn't collect as many documents, if they score too low.
> This doesn't make sense to me.  It should never be slower.  We're not 
> actually keeping track of any more hits, only stopping earlier.

Yes, that's puzzling... Please read below.

>> * Two-term Nutch queries result in complex Lucene BooleanQueries over 
>> many index fields, includng also PhraseQueries. These fared much 
>> worse than single-term queries: actually, the topN values were very 
>> low until MAX_HITS was increased to large values, and then all of a 
>> sudden all topN-s flipped into the 80-90% ranges.
> It would be interesting to try altering the generated query, to see if 
> it is the phrases or simply multiple terms which cause problems.  To 
> do this, one could hack the query-basic plugin, or simply alter query 
> boost parameters.  This 

I actually forgot to write that I don't use any of Nutch code. Early on 
I decided to eliminate this part in order to get first the raw 
performance from Lucene - but still using the Lucene queries 
corresponding to translated Nutch queries.

Please see the attached scripts. You will need to put bsh and lucene on 
your classpath, as well as compile and add the LimitingCollector classes 
(it was not possible to implement this inside bsh scripts, due to 
vagaries of class overloading in BeanShell). You need to customize the 
paths to both indexes inside the script. Then run:

    java -cp bsh.jar:lucene.jar:LimitedCollector.jar bsh.Interpreter "+url:information" 10000

The script will iterate 10 times for a warm-up, and then report the 
timing, and matching, and topN results. You will get an HTML table with 
the first 100 results.

> would help us figure out where the optimization is failing.  Suel used 
> multi-term queries, but not phrases, so we expect that the phrases are 
> causing the problem, but it would be good to see for certain.  We've 
> also never tuned Nutch's phrase matching, so it's also possible that 
> we may sometimes over-emphasize the phrase component in scores.  For 
> example, a slop of 10 might give better results and/or be more 
> amenable to this optimization.

In several installations I use smaller values of slop (around 20-40). 
But this is motivated by better quality matches, not by performance, so 
I didn't test for this...

>> I also noticed that the values of topN depended strongly on the 
>> document frequency of terms in the query. For a two-term query, where 
>> both terms have average document frequency, the topN values start 
>> from ~50% for low MAX_HITS. For a two-term query where one of the 
>> terms has a very high document frequency, the topN values start from 
>> 0% for low MAX_HITS. See the spreadsheet for details.
> Were these actually useful queries?  For example, I would not be 
> concerned if results differed greatly for a query like 'to be', since 
> that's not a very useful query.  Try searching for 'the the' on Google.

No, these were not stop-words. "High DF" is around 0.3, i.e. the term 
occurs in 30% documents; but it's still a valid and useful term.

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration  Contact: info at sigram dot com

View raw message