nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2039) Relevance based scoring filter
Date Mon, 15 Jun 2015 14:35:00 GMT


Sebastian Nagel commented on NUTCH-2039:

Thanks, [~sujenshah]! A few comments (I haven't tried to run the plugin):
* if SimilarityScoringFilter is inherited from AbstractScoringFilter it's not necessary to
implement all the "stub" methods (cf. NUTCH-1653)
* the gold-standard file needs to be parsed only once, right? This could be done from SimilarityScoringFilter.setConf(conf)
the resulting DocumentVector is then cached.
* If Nutch is run in Hadoop's distributed mode loading the gold-standard and stopwords file
should be done either from hdfs or from class path as for config files, e.g.
Reader reader = conf.getConfResourceAsReader(goldStandardFileName);

> Relevance based scoring filter
> ------------------------------
>                 Key: NUTCH-2039
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>            Reporter: Sujen Shah
>              Labels: memex, nutch
>             Fix For: 1.11
> A ScoringFilter plugin that uses a similarity measure to calculate the similarity between
a given page(gold standard) and the currently parsed page. The score obtained from this similarity
is then distributed to its outlinks. This filter aims to focus the crawler to crawl/explore
relevant pages. 

This message was sent by Atlassian JIRA

View raw message