nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Nioche (JIRA)" <j...@apache.org>
Subject [jira] [Created] (NUTCH-1075) Delegate language identification to Tika
Date Tue, 02 Aug 2011 06:20:27 GMT
Delegate language identification to Tika
----------------------------------------

                 Key: NUTCH-1075
                 URL: https://issues.apache.org/jira/browse/NUTCH-1075
             Project: Nutch
          Issue Type: Improvement
          Components: parser
    Affects Versions: 1.4
            Reporter: Julien Nioche
            Assignee: Julien Nioche
             Fix For: 1.4


In 2.0 the language identification is delegated to Tika and is done as part of the parsing
step (and not during the indexing as done currently).
The patch attached is a backport from trunk which implements this and adds a new parameter
to determine the strategy to use

{code:xml} 
<property>
  <name>lang.extraction.policy</name>
  <value>detect,identify</value>
  <description>This determines when the plugin uses detection and
  statistical identification mechanisms. The order in which the
  detect and identify are written will determine the extraction
  policy. Default case (detect,identify)  means the plugin will
  first try to extract language info from page headers and metadata,
  if this is not successful it will try using tika language
  identification. Possible values are:
    detect
    identify
    detect,identify
    identify,detect
  </description>
</property>
{code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message