lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven A Rowe" <sar...@syr.edu>
Subject FWD: RE: Fullwidth alphanumeric characters, plus a question on Korean ranges
Date Thu, 10 Jan 2008 22:18:45 GMT
Daniel Noll sent the message below addressed to me, and CC'd to java-dev.  I guess CC is not
good enough for ASF's mailing list software, since I received this message, but it never showed
up on the mailing list.  Belatedly forwarding it to the list now. - Steve

On 01/07/2008 at 5:06 PM, Daniel Noll wrote:
> -----Original Message-----
> From: Daniel Noll [mailto:daniel@nuix.com] 
> Sent: Monday, January 07, 2008 5:07 PM
> To: Steven A Rowe
> Cc: java-dev@lucene.apache.org
> Subject: Re: Fullwidth alphanumeric characters, plus a 
> question on Korean ranges
> 
> On Tuesday 08 January 2008 05:17:28 Steven A Rowe wrote:
> > Hi Daniel,
> > 
> > I think this discussion belongs on java-dev, so I'm replying there.
> > 
> > On 01/06/2008 at 7:47 PM, Daniel Noll wrote:
> > > We discovered [in StandardTokenizer.jj] that fullwidth letters are
> > > not treated as <LETTER> and fullwidth digits are not
> treated as <DIGIT>.
> > 
> > IMHO, this should be fixed in the JFlex version of StandardTokenizer -
> > do you have details?
> 
> The following ranges are relevant here:
> 
>   FF10-FF19  Fullwidth digits
>   FF21-FF3A  Fullwidth Latin uppercase
>   FF41-FF5A  Fullwidth Latin lowercase
>   
> > > Line 87:
> > >        "\uffa0"-"\uffdc"
> > > 
> > >   The halfwidth Katakana "letters" (as Unicode calls them) are in <CJ>
> > >   as expected, so I'm wondering if these halfwidth Hangul "letters"
> > >   should actually be in <KOREAN> instead of <LETTER>.
> > 
> > [U+FFA0-U+FFDC] is Hangul Jamo (phonetic symbols), not precomposed
> > Hangul syllables.
> 
> I know.  The Unicode spec just happens to call Jamo "letters".
> 
> > However, I just noticed that [U+1100-U+11FF] is included both in the
> > <LETTER> and <KOREAN> sections - not good.  I think [U+1100-U+11FF]
> > should be removed from the <LETTER> definition, and left as-is in the
> > <KOREAN> section; and [U+FFA0-U+FFDC] should be moved from <LETTER>
> to <KOREAN>.
> 
> I think so too.  I didn't notice this overlap... makes me
> wish the parser
> could detect character range overlaps and warn about them.
> 
> I had a bit more of a look through the Unicode blocks and
> found some more
> ranges which may or may not be worth considering.
> 
> These would seem to be worthy of going in <LETTER>:
>   2C00-2DDF  (multiple blocks which appear to contain more languages)
>   A720-A7FF  Latin Extended-D
>   A800-A82F  Syloti Nagri
>   A840-A87F  Phags-pa
> 
> There are these too, but they seem obscure...
>   2460-24FF  Enclosed Alphanumerics
> 
> Then you have ligatures, which if you use a normalising
> filter later may
> resolve to perfectly normal alphabetic characters:
>   FB00-FB4F  Alphabetic Presentation Forms
>   FB50-FBFF  Arabic Presentation Forms
> 
> Then we have some high extensions to CJK.  These are
> particularly interesting
> because they would be represented in UTF-16 as surrogates and
> I have no idea
> how to even add them to the grammar for that reason.
>   20000-2A6DF  CJK Unified Ideographs Extension B
>   2F800-2FA1F  CJK Compatibility Ideographs Supplement
> 
> There may be more hidden in the blocks which don't seem
> immediately obvious.
> 
> I wish the tokeniser could just use Character.isLetter and
> Character.isDigit instead of having to know all the ranges itself, since
> the JRE already has all this information.  Character.isLetter does
> return true for CJK characters though, so the ranges would still come in
> handy for determining what kind of letter they are.  I don't support
> JFlex has a way to do this...
> 
> Daniel
>

 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message