nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Joke (JIRA)" <>
Subject [jira] Commented: (NUTCH-548) Move URLNormalizer from Outlink to ParseOutputFormat
Date Tue, 04 Sep 2007 15:30:44 GMT


Emmanuel Joke commented on NUTCH-548:

Maybe I missed something but it seems we do it.

CrawlDb.update defined a JobConf which use CrawlDbFilter as Mapper.Class and set urlnomalizer
and filter. The urlnormalizer and filter flags are pass by the configuration ( i supposed
its when we set the plugin ).

Actually i find out while i was testing/debugging this patch, you can see it by running a
simple crawl in debug mode in Eclipse and set a debug breakpoint on RegexURLNormalizer.regexNormalizer.

You point an interesting thing. Why do we have a scope ? I tried to check the code and it
seems we never really use the scope defined in the function. Am i wrong ?

Beside if you look at the following codein regexNormalize:
  List curRules = (List)scopedRules.get(scope);
  if (curRules == null) {
     if (curRules == EMPTY_RULES || curRules == null) {
        LOG.warn("can't find rules for scope '" + scope + "', using default");
        scopedRules.put(scope, EMPTY_RULES);
    if (curRules == EMPTY_RULES || curRules == null) {
      // use global rules
      curRules = (List)scopedRules.get(URLNormalizers.SCOPE_DEFAULT);
Why don't we directly set the scopeRules defined for the SCOPE_DEFAULT for every scope which
has no rules instead of setting EMPTY_RULES and then getting the default rules ?

> Move URLNormalizer from Outlink to ParseOutputFormat
> ----------------------------------------------------
>                 Key: NUTCH-548
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>            Reporter: Emmanuel Joke
>            Assignee: Emmanuel Joke
>            Priority: Minor
>             Fix For: 1.0.0
>         Attachments: NUTCH-548.patch
> The idea is to avoid instantiating a new URLNormalizer for every OutLink. 
> So I move this operation to the ParseOutputFormat object.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message