nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jayesh Shende (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (NUTCH-2392) Get same pages multiple times if URL contains relative path
Date Thu, 08 Jun 2017 05:20:18 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jayesh Shende closed NUTCH-2392.
--------------------------------
    Resolution: Not A Bug

> Get same pages multiple times if URL contains relative path
> -----------------------------------------------------------
>
>                 Key: NUTCH-2392
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2392
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl
>    Affects Versions: 1.13
>         Environment: Ubuntu, JRE 1.8.131, Apache Solr 6.5.1
>            Reporter: Jayesh Shende
>            Priority: Critical
>              Labels: features
>             Fix For: 1.14
>
>   Original Estimate: 60h
>  Remaining Estimate: 60h
>
> When websites have relative URL at different pages for same HTML document, for example
on first depth I fetched contents of a page http://example.com/index.html, after few depths
I got a link (constructed by Nutch from some relative path pattern in some anchor tag) http://example.com/Level1/Level2/../../index.html
, in this case Nutch is fetching same HTML document two times considering both URLs are different
but they are not. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message