nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-1321) IDNNormalizer
Date Thu, 19 Dec 2013 17:38:07 GMT


Sebastian Nagel commented on NUTCH-1321:

Hi [~ilhamikalkan],
great! Thanks! The patch looks good (not tested yet). A few comments:
# method isPunycode(url)
String[] arr = url.split("\\.");
if (arr[1].startsWith("xn--"))
{code} fails for URLs like {{}}
# maybe we should make the decoding from Punycode to Unicode in scope indexer configurable
by some property "urlnormalizer.idn.indexer.decode" or similar. URLs are used as ordinary
content (tokenized field "url") and unique ID (field "id") for updating and deleting indexed
documents. Some indexer back-ends may require the id field to be pure ASCII or Punycode.
# cosmetics: code should be formatted by [eclipse-codeformat.xml|],
patches generated as decribed in [1|],

> IDNNormalizer
> -------------
>                 Key: NUTCH-1321
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>             Fix For: 1.9
>         Attachments: Nutch-1321.patch, idnNormalizer.patch
> Right now, IDN's are indexed as ASCII. An IDNNormalizer is to be used with an indexer
so it will encode ASCII URL's to their proper unicode equivalant.

This message was sent by Atlassian JIRA

View raw message