lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Sekiguchi (JIRA)" <>
Subject [jira] [Commented] (LUCENE-5674) A new token filter: SubSequence
Date Mon, 26 May 2014 13:41:01 GMT


Koji Sekiguchi commented on LUCENE-5674:

Didn't look at this, but I remember needing/writing something like this 10+ years ago....
but I think back then I wanted to have output be something like: com,,
- i.e. tokenized, but reversed order.

PathHierarchyTokenizer can tokenize something like that.

> A new token filter: SubSequence
> -------------------------------
>                 Key: LUCENE-5674
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/other
>            Reporter: Nitzan Shaked
>            Priority: Minor
>         Attachments: subseqfilter.patch
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> A new configurable token filter which, given a token breaks it into sub-parts and outputs
consecutive sub-sequences of those sub-parts.
> Useful for, for example, using during indexing to generate variations on domain names,
so that "" can be found by searching for "", or "".
> Parameters:
> sepRegexp: A regular expression used split incoming tokens into sub-parts.
> glue: A string used to concatenate sub-parts together when creating sub-sequences.
> minLen: Minimum length (in sub-parts) of output sub-sequences
> maxLen: Maximum length (in sub-parts) of output sub-sequences (0 for unlimited; negative
numbers for token length in sub-parts minus specified length)
> anchor: Anchor.START to output only prefixes, or Anchor.END to output only suffixes,
or Anchor.NONE to output any sub-sequence
> withOriginal: whether to output also the original token
> EDIT: now includes tests for filter and for factory.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message