nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Nutch Wiki] Update of "WhiteListRobots" by ChrisMattmann
Date Wed, 15 Apr 2015 14:56:48 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "WhiteListRobots" page has been changed by ChrisMattmann:
https://wiki.apache.org/nutch/WhiteListRobots

Comment:
- initial page

New page:
Nutch now has a [[https://issues.apache.org/jira/browse/NUTCH-1927|white list for robots.txt]]
capability that can be used to selectively on a per host and/or IP basis turn on/off robots.txt
parsing. Read on to find out how to use it.

Mime
View raw message