tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Sullivan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-856) Support CJK (Chinese, Japanese and Korean) language detection
Date Wed, 29 Feb 2012 00:27:01 GMT

    [ https://issues.apache.org/jira/browse/TIKA-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218751#comment-13218751
] 

James Sullivan commented on TIKA-856:
-------------------------------------

Actually, calling what I used a corpus may be overly generous. All I did was take the first
40,000 articles from the Japanese Wikipedia content and then removed the XML and non-Japanese
content (of which there was a lot given the nature of Wikipedia). 40,000 articles was enough
to get to the around 6,1 million that Jan H mentioned but it still was not working. I will
attach the generated ja.ngp.
                
> Support CJK (Chinese, Japanese and Korean) language detection
> -------------------------------------------------------------
>
>                 Key: TIKA-856
>                 URL: https://issues.apache.org/jira/browse/TIKA-856
>             Project: Tika
>          Issue Type: New Feature
>          Components: languageidentifier
>    Affects Versions: 1.0
>         Environment: All
>            Reporter: James Sullivan
>              Labels: Chinese, Japanese
>
> Support language detection of CJK (Chinese, Japanese and Korean).
> Some estimates have Chinese users overtaking English users on the Internet  so it is
important that these languages used by large number of people be supported.
> See TIKA-855

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message