nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Created] (NUTCH-2300) Fetcher to optionally save robots.txt
Date Fri, 19 Aug 2016 13:43:21 GMT
Sebastian Nagel created NUTCH-2300:

             Summary: Fetcher to optionally save robots.txt
                 Key: NUTCH-2300
             Project: Nutch
          Issue Type: Improvement
          Components: fetcher, protocol, segment
            Reporter: Sebastian Nagel
             Fix For: 1.13

For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response
(content and HTTP status). Of course, this should be optional and not by default.

This message was sent by Atlassian JIRA

View raw message