nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <j...@apache.org>
Subject [jira] Commented: (NUTCH-243) Some meta-refresh urls get ignored due to matching regular expression
Date Mon, 17 Mar 2008 16:50:24 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579507#action_12579507
] 

Andrzej Bialecki  commented on NUTCH-243:
-----------------------------------------

Duplicate of NUTCH-255 .

> Some meta-refresh urls get ignored due to matching regular expression
> ---------------------------------------------------------------------
>
>                 Key: NUTCH-243
>                 URL: https://issues.apache.org/jira/browse/NUTCH-243
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.8
>            Reporter: Dennis Kubes
>            Priority: Trivial
>
> On fetching of pages with meta-refresh tags the url is taken at face value without any
filtering.  Some urls, such as those used by struts return with a jsessionid or with query
strings.  Examples are:
> http://www.somesite.com;jsessionid=3123123412ADBE3344...
> http://www.somesite.com?querystring=value
> The RegexURLFilter will match these urls according to the following regex inside of the
regex-urlfilter.txt file:
> -[?*!@=]
> Should these urls be cleaned up to allow processing and not match the previous URL filter
or should they be ignored as they currently are?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message