lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] [Commented] (LUCENE-7960) NGram filters -- add option to keep short terms
Date Wed, 02 May 2018 22:24:00 GMT


Robert Muir commented on LUCENE-7960:

my biggest concern is that these filters would then have two ctors:

* NGramTokenFilter(TokenStream)
* NGramTokenFilter(TokenStream, int, int, boolean, boolean)

The no-arg one starts looking more attractive to users at this point, and its mega-trappy
(n=1,2)!!!!!!! That's the ctor that should be deprecated :)

In general I'll be honest, I don't like how trappy the apis are with these filters/tokenizers
because of defaults like that. I also think its trappy they take a min and a max at all, because
that's really creating (max-min) indexed fields all unioned into one. There aren't even any
warnings about this. 

I haven't reviewed what the booleans of the patch does, but I am concerned that the use case
may just be "keep original" which could be one boolean, or perhaps done in a different way
entirely (e.g. KeywordRepeatFilter or perhaps something like LUCENE-8273). So if its acceptable
to collapse it into one boolean that does that, I think that would be easier.

I feel like any defaults that our apis lead to (and when you have multiple ctors, then thats
a default) should be something that will perform and scale well and work for the general case.
For example n=4 has been shown to work well in many relevance experiments. At least we should
make it easy for you to explicitly ask for something like that without passing many parameters.

> NGram filters -- add option to keep short terms
> -----------------------------------------------
>                 Key: LUCENE-7960
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/analysis
>            Reporter: Shawn Heisey
>            Priority: Major
>         Attachments: LUCENE-7960.patch, LUCENE-7960.patch
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
> When ngram or edgengram filters are used, any terms that are shorter than the minGramSize
are completely removed from the token stream.
> This is probably 100% what was intended, but I've seen it cause a lot of problems for
users.  I am not suggesting that the default behavior be changed.  That would be far too disruptive
to the existing user base.
> I do think there should be a new boolean option, with a name like keepShortTerms, that
defaults to false, to allow the short terms to be preserved.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message