lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: TokenStream problem?
Date Fri, 18 Jul 2008 10:18:28 GMT

DM Smith wrote:

> According to the documentation for TokenStream, derived classes are  
> to override either next() or next(Token).
> Currently, if next(Token) is overridden, but next() is called,  
> payload is cloned if it exists in the new token.
> However, if next(Token) is called, it is up to the implementation to  
> properly maintain payload. It seems to me that since next(Token) is  
> called with a new Token(), that this should not be necessary.

I see: you mean that extra clone of the payload that we do in should not be necessary?  We added it for some  
reason -- oh it was done as part of LUCENE-1062.  I think the  
challenge is we can't be sure the TokenStream isn't sharing a single  
payload, even though we passed in a new Token, because the reuse API  
allows you to reuse Token and payload.  So it's safer to clone it  
because next() is supposed to provide a fully unshared Token.

> Also, if next() is overridden, but next(Token) is called, then  
> payload is never cloned, unless the overriding method handles it.

Well, in this case, if you override next(), you are supposed to return  
a fully unshared Token, so the payload should already be "private"?  I  


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message