nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2300) Fetcher to optionally save robots.txt
Date Fri, 19 Aug 2016 13:48:21 GMT


ASF GitHub Bot commented on NUTCH-2300:

GitHub user sebastian-nagel opened a pull request:

    NUTCH-2300 Fetcher to optionally save robots.txt

    If the property is set to true, Fetcher saves the robots.txt
    response (URL and Content including HTTP protocol status and metadata) in the
    segment (subfolder content/). It does not add a fetch datum, simply because this
    avoids that the robots.txt URL slips into CrawlDb or gets indexed. The robots.txt
    can then be retrieved from the segment, e.g., by
    # inject
    # generate
    # and fetch with
    bin/nutch fetch ...path_to_segment
    # dump segment (without -nocontent)
    bin/nutch readseg -dump ...path_to_segment ...path_to_dump
    cat ...path_to_dump/dump
    Version: -1
    contentType: text/html
    metadata: nutch.fetch.time=1471612087645 Server=Apache/2.4.7 (Ubuntu) Connection=close
Content-Length=208 Date=Fri, 19 Aug 2016 13:08:07 GMT Content-Type=text/html; charset=iso-8859-1

    <title>404 Not Found</title>
    <h1>Not Found</h1>
    <p>The requested URL /robots.txt was not found on this server.</p>

You can merge this pull request into a Git repository by running:

    $ git pull SaveRobotsTxt

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #141
commit 6c9cca5e55e43458cbc5e59b8591e4d27ac425a2
Author: Sebastian Nagel <>
Date:   2016-05-25T12:24:11Z

    Allow Fetcher to optionally store robots.txt content (if property
== true).
    Improved RobotRulesParser command-line tool.

commit 264eea01a4d868578dcf641d6ce405444d276929
Author: Sebastian Nagel <>
Date:   2016-08-19T13:06:14Z

    Ignore robots.txt when parsing segment, refactored storing of robots.txt in FetcherThread

commit 33cdca76ac91a63445d4e761081e8124a23413af
Author: Sebastian Nagel <>
Date:   2016-08-19T13:32:34Z

    add hint and log warning that works only in combination with


> Fetcher to optionally save robots.txt
> -------------------------------------
>                 Key: NUTCH-2300
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher, protocol, segment
>            Reporter: Sebastian Nagel
>             Fix For: 1.13
> For debugging or archival purposes it may be useful to let Fetcher store the robots.txt
response (content and HTTP status). Of course, this should be optional and not by default.

This message was sent by Atlassian JIRA

View raw message