lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: TokenStream and Token APIs
Date Wed, 15 Oct 2008 14:41:39 GMT

On Oct 11, 2008, at 7:33 PM, Michael Busch wrote:
> If a consumer is e. g. only interested in the positionIncrement and  
> Payload, it can consume the tokens like this:
>  public class Consumer {
>    public void consumeTokens(TokenStream ts) throws IOException {
>      Token token = ts.prototypeToken();
>      PayloadAttribute payloadSource = (PayloadAttribute) token;
>      PositionIncrementAttribute positionSource =
>                    (PositionIncrementAttribute) token;
>      while (ts.nextToken()) {
>        char[] term = token.termBuffer();
>        int termLength = token.termLength();
>        int positionIncrement = positionSource.getPositionIncrement();
>        Payload payload = payloadSource.getPayload();
>        // do something with the term, positionIncrement and payload
>      }
>    }
>  }

Bear with me, b/c I'm not sure I'm following, but looking at

, I see at least 5 different implemented Attributes.

So, let's say I add a 5 more attributes and now have a total of 10  
attributes.  Are you saying that I then would have, potentially, 10  
different variables that all point to the token as in the code snippet  
above where the casting takes place?  Or would I just create a single  
"Super" attribute that folds in all of my new attributes, plus any  
other existing ones?  Or, maybe, what I would do is create the 5 new  
attributes and then 1 new attribute that extends all 10, thus allowing  
me to use them individually, but saving me from having to do a whole  
ton of casting in my Consumer.

I like where this is going, but am still a bit fuzzy on how it's going  
to work in practice.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message