lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benson Margulies <ben...@basistech.com>
Subject Re: org.apache.lucene.analysis.icu.ICUNormalizer2Filter -- why Token?
Date Mon, 16 Sep 2013 17:41:26 GMT
Thanks, I might pitch in.


On Mon, Sep 16, 2013 at 12:58 PM, Robert Muir <rcmuir@gmail.com> wrote:

> Mostly because our tokenizers like StandardTokenizer will tokenize the
> same way regardless of normalization form or whether its normalized at
> all?
>
> But for other tokenizers, such a charfilter should be useful: there is
> a JIRA for it, but it has some unresolved issues
>
> https://issues.apache.org/jira/browse/LUCENE-4072
>
> On Sun, Sep 15, 2013 at 7:05 PM, Benson Margulies <bimargulies@gmail.com>
> wrote:
> > Can anyone shed light as to why this is a token filter and not a char
> > filter? I'm wishing for one of these _upstream_ of a tokenizer, so that
> the
> > tokenizer's lookups in its dictionaries are seeing normalized contents.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message