nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1403) Add default ScoringFilter for manipulating metadata
Date Sun, 30 Jun 2019 15:52:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875817#comment-16875817
] 

ASF GitHub Bot commented on NUTCH-1403:
---------------------------------------

aalbahem commented on pull request #458: fix for NUTCH-1403 contributed by aalbahem
URL: https://github.com/apache/nutch/pull/458
 
 
   The following were done: 
   1. Copy much of urlmeta to a new scoring plugin (scoring-metadata) 
   2. Introduce **scoring.db.md** to copy data from a crawl datum and inject to content 
   3. Introduce **scoring.content.md** to copy metadata from a content and inject to parse
data 
   4. Introduce **scoring.parse.md** to copy metadata form a parse data to a crawl datum outlinks
as db metadata 
   5. Change the necessary plugins to include the plugin in the clean, test and deploy tasks
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add default ScoringFilter for manipulating metadata 
> ----------------------------------------------------
>
>                 Key: NUTCH-1403
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1403
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Julien Nioche
>            Priority: Major
>
> This is currently done by the urlmeta plugin, which has too vague a name and a redundant
indexing filter now that we have the index-metadata plugin. This scoring filter would help
defining which metadata to pass from : 
> - the crawl metadata to the content metadata
> - the content metadata to the parse metadata
> - the parse metadata to the crawldatum for the outlinks
> I'd make this scoring filter available by default i.e. not in a separate plugin as its
functionalities are commonly used.   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message