nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Kubes <>
Subject Re: db.ignore.internal.links and ranking algorithms
Date Wed, 07 Nov 2007 22:53:52 GMT
I don't believe that you can set it for a single site. Setting the 
variable would affect all sites, but it should increase relevancy if you 
are doing web crawls and not constrained crawls.  This is because the 
default nutch scores internal and external links equally.

Are you merging segments or merging segment indexes?  Either way I don't 
think scores are recalculated.  Scores are calculated primarily on the 
parse process (distributing score to outlinks) and on the update db 
process (gathering inlink score and updating crawldatum).

Dennis Kubes

karthik085 wrote:
> Thanks for the quick reply. 
> Is there anyway I can set this score for one specific site? As I said
> earlier, I crawl a handful of sites - 1 site has lot of search results as
> they have high scores (many internal links and possibly, anchor pollution) -
> other site pages does not have many incoming internal links and anchor text
> are either useless or empty.
> At the end, I merge all the crawled segments into one for faster searching -
> won't the scores be recalculated here again? Setting the score for
> variable would then affect all sites. Won't it?
> Please correct me if I am wrong.
> Dennis Kubes-2 wrote:
>> Well, the short answer is it doesn't  Even if you set internal links to 
>> be ignored they are still calculated in the OPIC scoring and this 
>> negatively affects search relevancy.  The way to handle this is to set 
>> the variable to 0.0.  This way only external 
>> links are counted in OPIC.
>> I will post a wiki entry about this process soon.
>> Dennis Kubes
>> karthik085 wrote:
>>> Hi,
>>> I was wondering how does db.ignore.internal.links affect rankings on
>>> PageRank and OPIC algiorithm?  I searched on the forum - couldn't get a
>>> clear-cut answer.
>>> I am using Nutch 0.7.2 to crawl & index handful of sites. One site - has
>>> lot
>>> of pages and interlinks - around 1/3 of my total pages are from this site
>>> -
>>> hence, when I search for something and hit 'Show All Hits' - first 5-10
>>> pages are from this site - before any results from other sites are shown.
>>> How will db.ignore.internal.links help in this case?
>>> Of course, I will have to recrawl with nutch-0.9 to use OPIC
>>> algorithm...:-(
>>> Thanks.

View raw message