lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nitzan Shaked (JIRA)" <>
Subject [jira] [Created] (LUCENE-5674) A new token filter: SubSequence
Date Fri, 16 May 2014 11:24:44 GMT
Nitzan Shaked created LUCENE-5674:

             Summary: A new token filter: SubSequence
                 Key: LUCENE-5674
             Project: Lucene - Core
          Issue Type: Improvement
          Components: core/other
            Reporter: Nitzan Shaked
            Priority: Minor
         Attachments: 0001-SubSeqFilter.patch

A new configurable token filter which, given a token breaks it into sub-parts and outputs
consecutive sub-sequences of those sub-parts.

Useful for, for example, using during indexing to generate variations on domain names, so
that "" can be found by searching for "", or "".


sepRegexp: A regular expression used split incoming tokens into sub-parts.
glue: A string used to concatenate sub-parts together when creating sub-sequences.
minLen: Minimum length (in sub-parts) of output sub-sequences
maxLen: Maximum length (in sub-parts) of output sub-sequences (0 for unlimited; negative numbers
for token length in sub-parts minus specified length)
anchor: Anchor.START to output only prefixes, or Anchor.END to output only suffixes, or Anchor.NONE
to output any sub-sequence
withOriginal: whether to output also the original token

Note: tests will follow, I currently believe BaseTokenStreamTestCase is broken/mis-designed
for such use cases, I will open a separate Jira for that.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message