nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Meghna Kukreja" <>
Subject Bug in Nutch, possibly due to issues-273 and 322
Date Wed, 03 Jan 2007 19:03:43 GMT

I checked out the latest Nutch trunk to get the fix for Issues 273
( and 322
( and it looks like if
http.redirect.max is 0, then the redirected url is inserted with the
same status as the original url of CrawlDatum.STATUS_FETCH_REDIR_PERM
( line 201) which is converted to
CrawlDatum.STATUS_DB_REDIR_PERM. This prevents the redirected url from
being selected by the generator in the next round (
lines 140-142).
My initial seed url was a RSS feed where all the items were being
redirected. I tried to do a crawl with depth 3 but at the last depth
the Generator was not able to select any urls, printed the message "0
records selected for fetching, exiting ..." and the crawl ended with a
NullPointerException as the fetcher was trying to fetch an empty
I was wondering if any one else had seen this behaviour and has been
able to find a way to fix it.


View raw message