nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (NUTCH-2383) Wrong FS exception in Fetcher
Date Sat, 04 Nov 2017 16:43:00 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sebastian Nagel resolved NUTCH-2383.
------------------------------------
    Resolution: Not A Problem

Thanks [~yossi] for reporting this problem. Closing this as it can hardly be solved inside
Nutch: it's clear that the default value "local" of {{mapreduce.framework.name}} does not
allow to access hdfs:// paths. It's defined in [mapred-default.xml|https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml]
and should be set appropriately in mapred-site.xml which is not controlled by Nutch. It needs
to be configured when setting up the Hadoop cluster. Please reopen if you see any option to
fix this inside Nutch. Thanks!

> Wrong FS exception in Fetcher
> -----------------------------
>
>                 Key: NUTCH-2383
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2383
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.13
>         Environment: Hadoop 2.8 and Hadoop 2.7.2
>            Reporter: Yossi Tamari
>            Priority: Major
>         Attachments: crawl output.txt
>
>
> Running bin/crawl on either Hadoop 2.7.2 or Hadoop 2.8, the Injector and Generator succeed,
but the Fetcher throws: {code}java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9000/user/root/crawl/segments/20170430084337/crawl_fetch,
expected: file:///{code}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message