nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Nioche (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2012) Merge parsechecker and indexchecker
Date Fri, 15 May 2015 13:59:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545534#comment-14545534
] 

Julien Nioche commented on NUTCH-2012:
--------------------------------------

+1 to merging them into a more generic tool. Most of the code in these 2 classes is the same.
We could add a few options e.g. not to display the fields generated for the indexing

> Merge parsechecker and indexchecker
> -----------------------------------
>
>                 Key: NUTCH-2012
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2012
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.10
>            Reporter: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.11
>
>
> ParserChecker and IndexingFiltersChecker have evolved from simple tools to check parsers
and parsefilters resp. indexing filters to powerful tools which emulate the crawling of a
single URL/document:
> - check robots.txt (NUTCH-2002)
> - follow redirects (NUTCH-2004)
> Keeping both tools in sync takes extra work (cf. NUTCH-1757/NUTCH-2006, also NUTCH-2002,
NUTCH-2004 are done only for parsechecker). It's time to merge them
> * either into one general debugging tool, keeping parsechecker and indexchecker as aliases
> * centralize common code in one utility class



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message