nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki>
Subject Re: nutch is loosing not modified pages
Date Mon, 08 May 2006 07:48:37 GMT
Stefan Groschupf wrote:
> Hi,
> in the fetcher line 192 in case the status is NOTMODIFIED we collect  
> null as content but we already have the content.
> I'm worry what is happen with a page that does not change for 60 days, 
> since the concept of nutch is do delete segments that are older than 
> "db.default.fetch.interval", isn't it?
> If this is true, may be someone with write access can change null to 
> content.

This requires a more systematic approach, which is a part of the 
adaptive fetch patch. In that patch pages which are older than maximum 
fetch interval (a system-wide setting) will be forced on the fetchlist, 
no matter what their state. This also ensures that pages in the GONE 
state are checked from time to time.

I'll be working on this patch next week, with the goal of committing it, 
and I could use some testing and code review then ...

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration  Contact: info at sigram dot com

View raw message