nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (NUTCH-2172) index-more: document format of contenttype-mapping.txt
Date Sun, 06 Dec 2015 21:23:11 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sebastian Nagel resolved NUTCH-2172.
------------------------------------
    Resolution: Fixed

Committed to trunk, r1718223.

Thanks, [~nicola.tonellotto]! Although this solution is not that you've suggested, hopefully
it will help users to write the content type mappings in the right format. 
If a non-comment line does not follow the format the hadoop.log will show now a warning, e.g.:
{noformat}
2015-12-06 22:06:05,169 INFO  more.MoreIndexingFilter - Reading content type mappings from
file contenttype-mapping.txt
2015-12-06 22:06:05,174 WARN  more.MoreIndexingFilter - Wrong format of line: only spaces
no tabs
2015-12-06 22:06:05,174 WARN  more.MoreIndexingFilter - Expected format: <target type>
<tab> <type1> [<tab> <type2> ...]
{noformat}
Btw., multiple tabs as separators are no problem as the original format allows multiple content
types to be mapped to one target type. 

> index-more: document format of contenttype-mapping.txt
> ------------------------------------------------------
>
>                 Key: NUTCH-2172
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2172
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer, metadata
>    Affects Versions: 1.10
>         Environment: Macosx, Java 8
>            Reporter: Nicola Tonellotto
>            Assignee: Sebastian Nagel
>            Priority: Minor
>              Labels: easyfix, newbie
>             Fix For: 1.12
>
>         Attachments: NUTCH-2172-1.patch, NUTCH-2172-2.patch
>
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> The index-more plugin uses the conf/contenttype-mapping.txt file to build up the mimeMap
hash table (in the readConfiguration() method).
> The line splitting is performed around "\t", so it silently skip lines separated by simple
spaces or more than one tab (see line 325).
> Changing the single-char string "\t" with the regex "\\s+" should do the magic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message