lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-5202) LookaheadTokenFilter consumes an extra token in nextToken
Date Mon, 09 Sep 2013 16:35:54 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762011#comment-13762011
] 

Michael McCandless commented on LUCENE-5202:
--------------------------------------------

bq. I suspect that there's something that LTF does that I don't need that explains why it
is so complex.

I think it's trying to support arbitrary lookahead, and insertion of new tokens.  Sort of
what a SynonymFilter would need.

But it's obviously not easy to use yet :)
                
> LookaheadTokenFilter consumes an extra token in nextToken
> ---------------------------------------------------------
>
>                 Key: LUCENE-5202
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5202
>             Project: Lucene - Core
>          Issue Type: Bug
>    Affects Versions: 4.3.1
>            Reporter: Benson Margulies
>         Attachments: LUCENE-5202.patch, LUCENE-5202.patch
>
>
> This is a bit hard to explain except by looking at the test case. I've coded a filter
that uses LookaheadTokenFilter. The incrementToken method peeks some tokens. Then, it seems,
nextToken in the Lookahead class calls peekToken itself, which seems to me to consume a token
so that it's not seen when the derived class sets out to process the next set of tokens.
> In passing, this test case can be used to demonstrate that it does not work to try to
use the afterPosition method to set up attributes of the token that we're 'after'. Probably
that was never intended. However, I'm hoping for some feedback as to whether the rest of the
structure here is as intended for subclasses of LookaheadTokenFilter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message