nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jayesh Shende (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NUTCH-2392) Get same pages multiple times if URL contains relative path
Date Thu, 08 Jun 2017 05:19:18 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042214#comment-16042214
] 

Jayesh Shende edited comment on NUTCH-2392 at 6/8/17 5:18 AM:
--------------------------------------------------------------

The URLs I was working with were very long URLs with difference of one single letter cased
differently. Yes It is not a bug.


was (Author: jayesh):
The URLs I was worling with were very long URLs with difference of one single letter cased
differently. Yes It is not a bug.

> Get same pages multiple times if URL contains relative path
> -----------------------------------------------------------
>
>                 Key: NUTCH-2392
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2392
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl
>    Affects Versions: 1.13
>         Environment: Ubuntu, JRE 1.8.131, Apache Solr 6.5.1
>            Reporter: Jayesh Shende
>            Priority: Critical
>              Labels: features
>             Fix For: 1.14
>
>   Original Estimate: 60h
>  Remaining Estimate: 60h
>
> When websites have relative URL at different pages for same HTML document, for example
on first depth I fetched contents of a page http://example.com/index.html, after few depths
I got a link (constructed by Nutch from some relative path pattern in some anchor tag) http://example.com/Level1/Level2/../../index.html
, in this case Nutch is fetching same HTML document two times considering both URLs are different
but they are not. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message