nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Re: Bug in Nutch, possibly due to issues-273 and 322
Date Wed, 03 Jan 2007 19:50:38 GMT
Meghna Kukreja wrote:
> Hi,
>
> I checked out the latest Nutch trunk to get the fix for Issues 273
> (http://issues.apache.org/jira/browse/NUTCH-273) and 322
> (http://issues.apache.org/jira/browse/NUTCH-322) and it looks like if
> http.redirect.max is 0, then the redirected url is inserted with the
> same status as the original url of CrawlDatum.STATUS_FETCH_REDIR_PERM
> (Fetcher.java line 201) which is converted to
> CrawlDatum.STATUS_DB_REDIR_PERM. This prevents the redirected url from
> being selected by the generator in the next round (Generator.java
> lines 140-142).

Indeed, this looks wrong - the target of the redirect should be inserted 
with another fetch status (we could re-use status code LINKED, which is 
then turned into DB_UNFETCHED if a page didn't exist yet).

> My initial seed url was a RSS feed where all the items were being
> redirected. I tried to do a crawl with depth 3 but at the last depth
> the Generator was not able to select any urls, printed the message "0
> records selected for fetching, exiting ..." and the crawl ended with a
> NullPointerException as the fetcher was trying to fetch an empty
> segment.

Yup, another place to fix so that it doesn't throw an NPE.

> I was wondering if any one else had seen this behaviour and has been
> able to find a way to fix it.
Thanks for testing this, I'll apply these fixes soon.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Mime
View raw message