nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Nioche (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (NUTCH-1075) Delegate language identification to Tika
Date Tue, 02 Aug 2011 06:22:27 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche updated NUTCH-1075:
---------------------------------

    Attachment: NUTCH-1075.patch

Passes the tests but requires some testing

> Delegate language identification to Tika
> ----------------------------------------
>
>                 Key: NUTCH-1075
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1075
>             Project: Nutch
>          Issue Type: Improvement
>          Components: parser
>    Affects Versions: 1.4
>            Reporter: Julien Nioche
>            Assignee: Julien Nioche
>             Fix For: 1.4
>
>         Attachments: NUTCH-1075.patch
>
>
> In 2.0 the language identification is delegated to Tika and is done as part of the parsing
step (and not during the indexing as done currently).
> The patch attached is a backport from trunk which implements this and adds a new parameter
to determine the strategy to use
> {code:xml} 
> <property>
>   <name>lang.extraction.policy</name>
>   <value>detect,identify</value>
>   <description>This determines when the plugin uses detection and
>   statistical identification mechanisms. The order in which the
>   detect and identify are written will determine the extraction
>   policy. Default case (detect,identify)  means the plugin will
>   first try to extract language info from page headers and metadata,
>   if this is not successful it will try using tika language
>   identification. Possible values are:
>     detect
>     identify
>     detect,identify
>     identify,detect
>   </description>
> </property>
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message