nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2032) Plugin to index the raw content of a readable document.
Date Mon, 08 Jun 2015 09:45:01 GMT


Sebastian Nagel commented on NUTCH-2032:

Hi [~betolink], from the point of "intrusiveness" or encapsulation I agree - it would be better
to have all necessary functionality bundled inside a single plugin. But in doubt performance
is more important, esp. at scale. With regard to indexing raw content these points may impact
* increasing the size of segments by adding raw content twice (in average by 50% or more)
* reading unnecessarily the raw content from segments if raw content isn't indexed (most users
only want the readable text to be indexed)

The last patch for NUTCH-1785 does not impact performance. The solution via a plugin index-rawcontent
Btw., in 2.x plugins can request via the FieldPluggable interface which fields are read from
the web table. That's nice, but in 1.x we don't have such a functionality.

> Plugin to index the raw content of a readable document. 
> --------------------------------------------------------
>                 Key: NUTCH-2032
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: indexer, parser
>    Affects Versions: 1.10
>            Reporter: Luis Lopez
>              Labels: content, index, index-rawcontent, parser, raw
>             Fix For: 1.11
> This is related to and 
> We created a couple plugins to index the raw content of readable documents. If we include
these plugins in the plugin chain we'll index the raw content of a readable document, i.e.
XML, HTML, CSV, TXT etc. The index-rawcontent plugin is not designed to index binary files,
however having the full content of an HTML/XML or a CSV document is really critical for some
of us.

This message was sent by Atlassian JIRA

View raw message