nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2035) Regex filter using case sensitive rules.
Date Wed, 03 Jun 2015 21:27:38 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571680#comment-14571680
] 

Sebastian Nagel commented on NUTCH-2035:
----------------------------------------

Thanks, [~betolink]!
* is it possible to substantiate "computationally expensive"? What CPU time is saved by the
patch for which Java version, URL sample, etc.
* I've run a quick test using 100k URLs and
{noformat}
time $NUTCH_HOME/bin/nutch plugin urlfilter-regex org.apache.nutch.urlfilter.regex.RegexURLFilter
<100k_urls.txt >/dev/null
{noformat}
with 3 runs per config/test and taking the shortest time for each configuration got only minimal
savings for the patched trial: 11.94 sec. vs 11.97 (user time). That's not really significant
(agreed, my trivial measurement isn't significant, too).
* currently the user is able to decide whether to match case sensitive or not. Shall we really
remove this and make the regex url filter be always case insensitive?
* if it's about readability of the patterns, why not change the pattern to:
{noformat}
-(?i)\.(gif|jpg|png|ico|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|exe|jpeg|bmp|js)$
{noformat}
Btw., this variant seems to win the race: 11.52 sec.
* in general, the plugin urlfilter-suffix is much faster to filter by suffix patterns, esp.
if the list gets longer. There is already a hint in to conf/regex-urlfilter.txt.template
 regarding this point. 

> Regex filter using case sensitive rules.
> ----------------------------------------
>
>                 Key: NUTCH-2035
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2035
>             Project: Nutch
>          Issue Type: Improvement
>          Components: plugin
>    Affects Versions: 1.10
>            Reporter: Luis Lopez
>            Priority: Minor
>              Labels: filters, regex, regex-urlfilter
>             Fix For: 1.11
>
>
> Regex expressions are computationally expensive and having “EXE|exe|JPG|jpg” etc
etc..... adds up if we use complex rules.
> Regex filter should use case insensitive rules to make the rules more readable and improve
performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message