nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Canan Girgin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1314) Impose a limit on the length of outlink target urls
Date Mon, 08 Feb 2016 08:45:39 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136709#comment-15136709
] 

Canan Girgin commented on NUTCH-1314:
-------------------------------------

[~lewismc], Please could somebody help me commit  NUTCH-1314-v4.patch If there is no problem
?

> Impose a limit on the length of outlink target urls
> ---------------------------------------------------
>
>                 Key: NUTCH-1314
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1314
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Ferdy Galema
>             Fix For: 2.4, 1.12
>
>         Attachments: NUTCH-1314-trunk.patch, NUTCH-1314-v2.patch, NUTCH-1314-v3.patch,
NUTCH-1314-v4.patch, NUTCH-1314.patch
>
>
> In the past we have encountered situations where crawling specific broken sites resulted
in ridiciously long urls that caused the stalling of tasks. The regex plugins (normalizing/filtering)
processed single urls for hours, if not indefinitely hanging.
> My suggestion is to limit the outlink url target length as soon possible. It is a configurable
limit, the default is 3000. This should be reasonably long enough for most uses. But sufficienly
strict enough to make sure regex plugins do not choke on urls that are too long. Please see
attached patch for the Nutchgora implementation.
> I'd like to hear what you think about this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message