nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2032) Plugin to index the raw content of a readable document.
Date Mon, 08 Jun 2015 09:45:01 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576839#comment-14576839
] 

Sebastian Nagel commented on NUTCH-2032:
----------------------------------------

Hi [~betolink], from the point of "intrusiveness" or encapsulation I agree - it would be better
to have all necessary functionality bundled inside a single plugin. But in doubt performance
is more important, esp. at scale. With regard to indexing raw content these points may impact
performance:
* increasing the size of segments by adding raw content twice (in average by 50% or more)
* reading unnecessarily the raw content from segments if raw content isn't indexed (most users
only want the readable text to be indexed)

The last patch for NUTCH-1785 does not impact performance. The solution via a plugin index-rawcontent
does.
Btw., in 2.x plugins can request via the FieldPluggable interface which fields are read from
the web table. That's nice, but in 1.x we don't have such a functionality.

> Plugin to index the raw content of a readable document. 
> --------------------------------------------------------
>
>                 Key: NUTCH-2032
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2032
>             Project: Nutch
>          Issue Type: New Feature
>          Components: indexer, parser
>    Affects Versions: 1.10
>            Reporter: Luis Lopez
>              Labels: content, index, index-rawcontent, parser, raw
>             Fix For: 1.11
>
>
> This is related to https://issues.apache.org/jira/browse/NUTCH-1785 and 
> https://issues.apache.org/jira/browse/NUTCH-1458
> We created a couple plugins to index the raw content of readable documents. If we include
these plugins in the plugin chain we'll index the raw content of a readable document, i.e.
XML, HTML, CSV, TXT etc. The index-rawcontent plugin is not designed to index binary files,
however having the full content of an HTML/XML or a CSV document is really critical for some
of us.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message