nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Re: IndexSorter optimizer
Date Mon, 02 Jan 2006 22:49:40 GMT
Doug Cutting wrote:

> Andrzej Bialecki wrote:
>
>> Using the original index, it was possible for pages with high tf/idf 
>> of a term, but with a low "boost" value (the OPIC score), to outrank 
>> pages with high "boost" but lower tf/idf of a term. This phenomenon 
>> leads quite often to results that are perceived as "junk", e.g. pages 
>> with a lot of repeated terms, but with little other real content, 
>> like for example navigation bars.
>
>
> Sounds like tf/idf might be de-emphasized in scoring.  Perhaps 
> NutchSimilarity.tf() should use log() instead of sqrt() when 
> field==content?


I don't think it's that simple, the OPIC score is what determined this 
behaviour, and it doesn't correspond at all to tf/idf, but to a human 
judgement.

>
>> To conclude, I will add the IndexSorter.java to the core classes, and 
>> I suggest to continue the experiments ...
>
>
> I've updated the version of Lucene included with Nutch to have the 
> required patch.  Would you like me to commit IndexSorter.java or would 
> you?


Please do it. There are two typos in your version of IndexSorter, you 
used numDocs() in two places instead of maxDoc(), which for indexes with 
deleted docs (after dedup) leads to exceptions.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Mime
View raw message