nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <>
Subject [jira] Commented: (NUTCH-753) Prevent new Fetcher to retrieve the robots twice
Date Sat, 28 Nov 2009 14:09:20 GMT


Hudson commented on NUTCH-753:

Integrated in Nutch-trunk #995 (See [])
     Prevent new Fetcher from retrieving the robots twice.

> Prevent new Fetcher to retrieve the robots twice
> ------------------------------------------------
>                 Key: NUTCH-753
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Julien Nioche
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.1
>         Attachments: NUTCH-753.patch
> The new Fetcher which is now used by default handles the robots file directly instead
of relying on the protocol. The options Protocol.CHECK_BLOCKING and Protocol.CHECK_ROBOTS
are set to false to prevent fetching the robots.txt twice (in Fetcher + in protocol), which
avoids calling robots.isAllowed. However in practice the robots file is still fetched as there
is a call to robots.getCrawlDelay() a bit further which is not covered by the if (Protocol.CHECK_ROBOTS).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message