nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2581) Caching of redirected robots.txt may overwrite correct robots.txt rules
Date Wed, 06 Jun 2018 08:03:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502949#comment-16502949
] 

ASF GitHub Bot commented on NUTCH-2581:
---------------------------------------

sebastian-nagel opened a new pull request #342: NUTCH-2581 Caching of redirected robots.txt
may overwrite correct robots.txt rules (fix for 2.x)
URL: https://github.com/apache/nutch/pull/342
 
 
   - only cache redirected robots.txt rules if the target URL path and
     query equal /robots.txt

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Caching of redirected robots.txt may overwrite correct robots.txt rules
> -----------------------------------------------------------------------
>
>                 Key: NUTCH-2581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2581
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher, robots
>    Affects Versions: 2.3.1, 1.14
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Critical
>             Fix For: 2.4, 1.15
>
>
> Redirected robots.txt rules are also cached for the target host. That may cause that
the correct robots.txt rules are never fetched. E.g., http://wyomingtheband.com/robots.txt
redirects to https://www.facebook.com/wyomingtheband/robots.txt. Because fetching fails with
a 404 bots are allowed to crawl wyomingtheband.com. The rules is erroneously also cached for
the redirect target host www.facebook.com which is clear regarding their [robots.txt|https://www.facebook.com/robots.txt]
rules and does not allow crawling.
> Nutch may cache redirected robots.txt rules only if the path part (in doubt, including
the query) of the redirect target URL is exactly {{/robots.txt}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message