lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ingomar Wesp (JIRA)" <>
Subject [jira] [Commented] (LUCENE-7960) NGram filters -- add option to keep short terms
Date Thu, 03 May 2018 06:20:00 GMT


Ingomar Wesp commented on LUCENE-7960:

I understand your concern. As far as I can tell, there are two options:

1) Replace the booleans with an enum that covers the four possible combinations. Maybe "keepMode"
with values "DROP", "KEEP_SHORT_TERM", "KEEP_LONG_TERM", "KEEP_ALL". I'm not really happy
with the names, though - advice welcome.

2) Fold the two booleans into one - maybe "preserveOriginal", akin to how the corresponding
attribute in other filters is called.

I personally prefer 1), but I'd happily adapt the patch to implement the other if it makes
things easier from your perspective.

> NGram filters -- add option to keep short terms
> -----------------------------------------------
>                 Key: LUCENE-7960
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/analysis
>            Reporter: Shawn Heisey
>            Priority: Major
>         Attachments: LUCENE-7960.patch, LUCENE-7960.patch
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
> When ngram or edgengram filters are used, any terms that are shorter than the minGramSize
are completely removed from the token stream.
> This is probably 100% what was intended, but I've seen it cause a lot of problems for
users.  I am not suggesting that the default behavior be changed.  That would be far too disruptive
to the existing user base.
> I do think there should be a new boolean option, with a name like keepShortTerms, that
defaults to false, to allow the short terms to be preserved.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message