nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@nutch.org>
Subject Re: IndexOptimizer (Re: Lucene performance bottlenecks)
Date Thu, 15 Dec 2005 18:00:13 GMT
Andrzej Bialecki wrote:
> Doug Cutting wrote:
>> The graph just shows that they differ, not how much better or worse 
>> they are, since the baseline is not perfect.  When the top-10 is 50% 
>> different, are those 5 different hits markedly worse matches to your 
>> eye than the five they've displaced, or are they comparable?  That's 
>> what really matters.
> 
> Hmm. I'm not sure I agree with this. Your reasoning would be true if we 
> were changing the ranking formula. But the goal IMHO with these patches 
> is to return equally complete results, using the same ranking formula.

But we should not assume that the ranking formula is perfect.  Imagine a 
case where the high-order bits of the score are correct and the 
low-order bits are random.  Then an optimization which changes local 
orderings does not actually affect result quality.

> I specifically avoided using normalized scores, instead using the 
> absolute scores in TopDocs. And the absolute scores in both cases are 
> exactly the same, for those results that are present.
> 
> What is wrong is that some results that should be there (judging by the 
> ranking) are simply missing. So, it's about the recall, and the baseline 
> index gives the best estimate.

Yes, this optimization, by definition, hurts recall.  The only question 
is does it substantially hurt relevance at, e.g., 10 hits.  If the 
top-10 are identical then the answer is easy: no, it does not.  But if 
they differ, we can only answer this by looking at results.  Chances are 
they're worse, but how much?  Radically?  Slightly?  Noticiably?

>> What part of Nutch are you trying to avoid?  Perhaps you could try 
>> measuring your Lucene-only benchmark against a Nutch-based one.  If 
>> they don't differ markedly then you can simply use Nutch, which makes 
>> it a stronger benchmark.  If they differ, then we should figure out why.
> 
> Again, I don't see it this way. Nutch results will always be worse than 
> pure Lucene, because of the added layers. If I can't improve the 
> performance in Lucene code (which takes > 85% time for every query) then 
> no matter how well optimized Nutch code is it won't get far.

But we're mostly modifying Nutch's use of Lucene, not modifying Lucene. 
  So measuring Lucene alone won't tell you everything, and you'll keep 
having to port Nutch stuff.  If you want to, e.g., replay a large query 
log to measure average performance, then you'll need things like 
auto-filterization, n-grams, query plugins, etc., no?

>>> In several installations I use smaller values of slop (around 20-40). 
>>> But this is motivated by better quality matches, not by performance, 
>>> so I didn't test for this...
>>
>> But that's a great reason to test for it!  If lower slop can improve 
>> result quality, then we should certainly see if it also makes 
>> optimizations easier.
> 
> I forgot to mention this - the tests I ran already used the smaller 
> values: the slop was set to 20.

Are they different if the slop is Integer.MAX_VALUE?  It would be really 
good to determine what causes results to diverge, whether it is multiple 
terms (probably not) phrases (probably) and/or slop (perhaps).  Chances 
are that the divergence is bad, that results are adversely affected, and 
that we need to try to fix it.  But to do so we'll need to understand it.

> That's another advantage of using Lucene directly in this script - you 
> can provide any query structure on the command-line without changing the 
> code in Nutch.

But that just means that we should set the SLOP constant in 
BasicQueryFilter.java from a configuration property, and permit the 
setting of configuration properties from the command line, no?

Doug

Mime
View raw message