nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-706) Url regex normalizer
Date Tue, 02 Oct 2012 19:19:08 GMT


Sebastian Nagel commented on NUTCH-706:

Are there objections to apply and commit the patch? Tests pass for both trunk and 2.x.
The problem is reported twice. Until there is a more sophisticated URL normalizer (see Ken
Krugler's comment) there is no real alternative then improving the regex pattern.

> Url regex normalizer
> --------------------
>                 Key: NUTCH-706
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Meghna Kukreja
>            Priority: Minor
>             Fix For: 1.6
>         Attachments: NUTCH-706-2.patch, NUTCH-706.patch
> Hey,
> I encountered the following problem while trying to crawl a site using
> nutch-trunk. In the file regex-normalize.xml, the following regex is
> used to remove session ids:
> <pattern>([;_]?((?i)l|j|bv_)?((?i)sid|phpsessid|sessionid)=.*?)(\?|&amp;|#|$)</pattern>.
> This pattern also transforms a url, such as,
> "&newsId=2000484784794&newsLang=en" into "&new&newsLang=en" (since it
> matches 'sId' in the 'newsId'), which is incorrect and hence does not
> get fetched. This expression needs to be changed to prevent this.
> Thanks,
> Meghna

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message