lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DM Smith <>
Subject Re: Token implementation
Date Tue, 20 May 2008 09:12:04 GMT

On May 20, 2008, at 12:50 AM, Hiroaki Kawai wrote:

> "Michael McCandless" <> wrote:

>> More responses below:
>> DM Smith <> wrote:

>>> But, in TokenFilter, next() should be deprecated, IMHO.
>> I think this is a good idea.  After all if people don't want to  
>> bother
>> using the passed in Token, they are still allowed to return a new
>> one.
> I could not see what you meant. Can I ask you to let me know the  
> reason
> why it should be deprecated?

The purpose of the char[] rather than a String is to promote reuse of  
a mutable buffer. Reusing a Token minimizes the number of  
constructions. Each TokenFilter has both
Token next()
Token next(Token)

If next() is implemented then he TokenFilter does not have access to a  
shared Token. While next(Token) supplies a shared token by the caller.

The proper way to use next(Token) is as follows

t =;

can be implemented as
Token next(Token result) {
   Token t = new Token();
   return t;

This would be the trivial equivalent to next(). Just ignore the  
argument and do as before.

-- DM

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message