nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2035) Regex filter using case sensitive rules.
Date Wed, 03 Jun 2015 21:27:38 GMT


Sebastian Nagel commented on NUTCH-2035:

Thanks, [~betolink]!
* is it possible to substantiate "computationally expensive"? What CPU time is saved by the
patch for which Java version, URL sample, etc.
* I've run a quick test using 100k URLs and
time $NUTCH_HOME/bin/nutch plugin urlfilter-regex org.apache.nutch.urlfilter.regex.RegexURLFilter
<100k_urls.txt >/dev/null
with 3 runs per config/test and taking the shortest time for each configuration got only minimal
savings for the patched trial: 11.94 sec. vs 11.97 (user time). That's not really significant
(agreed, my trivial measurement isn't significant, too).
* currently the user is able to decide whether to match case sensitive or not. Shall we really
remove this and make the regex url filter be always case insensitive?
* if it's about readability of the patterns, why not change the pattern to:
Btw., this variant seems to win the race: 11.52 sec.
* in general, the plugin urlfilter-suffix is much faster to filter by suffix patterns, esp.
if the list gets longer. There is already a hint in to conf/regex-urlfilter.txt.template
 regarding this point. 

> Regex filter using case sensitive rules.
> ----------------------------------------
>                 Key: NUTCH-2035
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: plugin
>    Affects Versions: 1.10
>            Reporter: Luis Lopez
>            Priority: Minor
>              Labels: filters, regex, regex-urlfilter
>             Fix For: 1.11
> Regex expressions are computationally expensive and having “EXE|exe|JPG|jpg” etc
etc..... adds up if we use complex rules.
> Regex filter should use case insensitive rules to make the rules more readable and improve

This message was sent by Atlassian JIRA

View raw message