nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1319) HostNormalizer
Date Fri, 15 Jun 2012 17:54:43 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295804#comment-13295804
] 

Hudson commented on NUTCH-1319:
-------------------------------

Integrated in Nutch-trunk #1869 (See [https://builds.apache.org/job/Nutch-trunk/1869/])
    NUTCH-1319 HostNormalizer plugin (Revision 1349236)

     Result = SUCCESS
markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349236
Files : 
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/host-urlnormalizer.txt
* /nutch/trunk/src/plugin/urlnormalizer-host
* /nutch/trunk/src/plugin/urlnormalizer-host/build.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/data
* /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt
* /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/src
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java

                
> HostNormalizer
> --------------
>
>                 Key: NUTCH-1319
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1319
>             Project: Nutch
>          Issue Type: New Feature
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>             Fix For: 1.6
>
>         Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a given host
to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website
is one of many on the internet that has a duplicate website on the same domain just because
it allows both www and non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if there are
multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these problems. It also
leads to skewed scores in the webgraph as different websites link to different versions of
the same duplicate website.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message