nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2690) Configurable and fast URL filter
Date Mon, 06 May 2019 15:32:00 GMT


ASF GitHub Bot commented on NUTCH-2690:

sebastian-nagel commented on pull request #433: NUTCH-2690 Configurable and fast URL filter
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Configurable and fast URL filter
> --------------------------------
>                 Key: NUTCH-2690
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: plugin
>            Reporter: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.16
> This improvement introduces a new URL filter plugin "urlfilter-fast" (naming debatable)
which is in use at Common Crawl [since 2013|]
to apply a long list of filters. 
> # an exact (suffix) match against the host name is done to retrieve host/domain-specific
regex rules
> # applies a regular expression against the path (and query) component of the URL
> What makes it faster than urlfilter-regex for common cases:
> - regexes are selected by host name or it's domain suffix, so there are usually fewer
rules to be checked. That's similar to NUTCH-1838 but any domain suffix can be matched including
{{}}, {{com}} or {{.}} for global rules. The selection by host name suffix
is considerably fast.
> - regexes are applied only to the path component (optionally including the query) and
not the entire URL.
>   Matching against a shorter string can make a huge difference for more complex regular
> - the rule to deny everything from a host or domain gets special treatment to be fast
> More details about the rule format are found in the plugin's [README|].

This message was sent by Atlassian JIRA

View raw message