nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rogério Pereira Araújo (JIRA) <>
Subject [jira] [Commented] (NUTCH-1483) Can't crawl filesystem with protocol-file plugin
Date Thu, 01 Nov 2012 01:03:12 GMT


Rogério Pereira Araújo commented on NUTCH-1483:

One important thing to mention, if I add the following regex to regex-urlfilter:


No documents gets crawled, in order to get documents crawled I must leave the default which

> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>                 Key: NUTCH-1483
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: protocol
>    Affects Versions: 1.6, 2.1
>         Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
>            Reporter: Rogério Pereira Araújo
>         Attachments: NUTCH-1483.patch
> I tried to follow the same steps described in this wiki page:
> I made all required changes on regex-urlfilter.txt and added the following entry in my
seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch
has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
>         at org.apache.nutch.protocol.file.File.getProtocolOutput(
>         at org.apache.nutch.fetcher.FetcherReducer$
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError:
File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/
> Note: The regex-urlfilter entry only works as expected if I add the entry 
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message