nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2002) ParserChecker to check robots.txt
Date Fri, 15 May 2015 13:58:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545532#comment-14545532
] 

Sebastian Nagel commented on NUTCH-2002:
----------------------------------------

one point: also redirects should be checked for robots.txt (after NUTCH-2004)

> ParserChecker to check robots.txt
> ---------------------------------
>
>                 Key: NUTCH-2002
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2002
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.9
>            Reporter: Julien Nioche
>            Priority: Minor
>         Attachments: NUTCH-2002.patch
>
>
> ParserChecker could check whether a given URL is allowed by the robots.txt directives.
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message