lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rowe (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-478) CJK char list
Date Thu, 05 Jan 2006 00:58:02 GMT
    [ http://issues.apache.org/jira/browse/LUCENE-478?page=comments#action_12361804 ] 

Steven Rowe commented on LUCENE-478:
------------------------------------

There are six classes of issues:

1. A character range in StandardTokenizer.jj that is missing in
John's list, and should be left as-is in StandardTokenizer.jj
(in the <CJ> section):

   1.a. [ U+3100 - U+312F ]
        BoPoMoFo (a.k.a. ZhuYin): Phonetic transcription symbols
        used in Taiwan; not used on mainland China.

2. A character range in StandardTokenizer.jj that is also in
John's list, but in the <LETTER> section rather than in the <CJ>
section, and should be left as-is:

   2.a. [ U+1100 - U+11FF ]
        Korean Jamo (phonetic symbols)

3. A character range in StandardTokenizer.jj that is not present in
John's list, and that should be removed from the <KOREAN> section
in StandardTokenizer.jj:

   3.a. [ U+D7A4 - U+D7AF ]
        Non-character range at the end of the pre-composed Hangul 
        (Korean) block

4. A character range in John's list that is missing in
StandardTokenizer.jj, but which was not present in Unicode 3.0, and
so strictly should not be included when running on Java 1.4; since
this is a non-character range in Unicode 3.0, however, I think it
should be included in StandardTokenizer.jj (in the <CJ> section)
for future compatibility with Java 1.5 and Unicode 4.0:

   4.a. [ U+31F0 - U+31FF ]
        Japanese Katakana phonetic extensions; these were introduced
        in Unicode version 3.2 (see
        http://www.unicode.org/reports/tr28/tr28-3.html#10_3_katakana )

5. Character ranges in John's list that are missing in
StandardTokenizer.jj, and that should be added to the newly
re-labeled <CJ> section:

   5.a. [ U+FF65 - U+FF9F ]
        Half-width Japanese Katakana (phonetic symbols)
   
   5.b. [ U+3d2e - U+4DB5 ] (non-chars [ U+4DB6 - U+4DBF ] excluded)
        CJK Ideograph Extension A.  
        This range was introduced in Unicode 3.0.

6. A character range in John's list that is missing in
StandardTokenizer.jj, and that should be added to the <LETTER>
section, since it, like the [ U+1100 - U+11FF ] range already
included there, is a range of Korean Jamo (phonetic symbols):

   6.a. [ U+FFA0 - U+FFDC ]
        Half-width Korean Jamo (phonetic symbols)


> CJK char list
> -------------
>
>          Key: LUCENE-478
>          URL: http://issues.apache.org/jira/browse/LUCENE-478
>      Project: Lucene - Java
>         Type: Bug
>   Components: Analysis
>     Versions: 1.4
>     Reporter: John Wang
>     Priority: Minor

>
> Seems the character list in the CJK section of the StandardTokenizer.jj is not quite
complete. Following is a more complete list:
> < CJK:                                          // non-alphabets
>       [
> 	   "\u1100"-"\u11ff",
>        "\u3040"-"\u30ff",
>        "\u3130"-"\u318f",
>        "\u31f0"-"\u31ff",
>        "\u3300"-"\u337f",
>        "\u3400"-"\u4dbf",
>        "\u4e00"-"\u9fff",
>        "\uac00"-"\ud7a3",
>        "\uf900"-"\ufaff",
>        "\uff65"-"\uffdc"       
>       ]
>   >

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message