nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Assigned] (NUTCH-2683) DeduplicationJob: add option to prefer https:// over http://
Date Wed, 10 Apr 2019 11:35:00 GMT


Sebastian Nagel reassigned NUTCH-2683:

    Assignee: Sebastian Nagel

> DeduplicationJob: add option to prefer https:// over http://
> ------------------------------------------------------------
>                 Key: NUTCH-2683
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: crawldb
>    Affects Versions: 1.15
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Major
>             Fix For: 1.16
> The deduplication job allows to keep the shortest URLs as the "best" URL of a set of
duplicates, marking all longer ones as duplicates. Recently search engines started to penalize
non-https pages by [giving https pages a higher rank|]
and [marking http as insecure|].
> If URLs are identical except for the protocol the deduplication job should be able to
prefer https:// over http:// URLs, although the latter ones are shorter by one character.
Of course, this should be configurable and in addition to existing preferences (length, score
and fetch time) to select the "best" URL among duplicates.

This message was sent by Atlassian JIRA

View raw message