lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Busch <>
Subject Re: New Token API was Re: Payloads and TrieRangeQuery
Date Mon, 15 Jun 2009 00:05:00 GMT
On 6/14/09 5:17 AM, Grant Ingersoll wrote:
> Agreed.  I've been bringing it up for a while now and made the same 
> comments when it was first introduced, but felt like the lone voice in 
> the wilderness on it and gave way [1], [2], [3].  Now that others are 
> writing/converting, I think it is worth revisiting.

I am and always was open to constructive suggestions about how to design 
this API.  I know these new APIs currently don't seem to have many 
advantages over the previous ones, but they're basically laying the API 
groundwork for future features like flexible indexing. Some concerns you 
mentioned were targeted against the first version of the patch in 
LUCENE-1422. But, you later said you liked how the next patch looked (in 
thread [2] that you mentioned).

> That being said, I did just write my first TokenFilter with it, and 
> didn't think it was that hard.  There are some gains in it and the API 
> can be simpler if you just need one or two attributes (see 
> DelimitedPayloadTokenFilter), although, just like the move to using 
> char [] in Token, as soon as you do something like store a Token, you 
> lose most of the benefit, I think (for the char [] case, as soon as 
> you need a String in one of your filters, you lose the perf. gain).  
> The annoying parts are that you still have to implement the deprecated 
> next() part, otherwise chances are the thing is unusable by everyone 
> at this point anyway.
I'm not sure why this (currently having to implement next() too) is such 
an issue for you. You brought it up at the Lucene meetup too. No user 
will ever have to implement both (the new API and the old) in their 
streams/filters. The only reason why we did it this way is to not 
sacrifice performance for existing streams/filters when people switch to 
Lucene 2.9. I explained this point in the jira issue:

The only time when we'll ever have to implement both APIs is between now 
and 2.9, only for new streams and filters that we add before 2.9 is 
released. I don't think it'd be reasonable to consider this disadvantage 
as a show stopper.
> Add on top of it, that the whole point of customizing the chain is to 
> use it in search and, frankly speaking, somehow I think that part of 
> the patch was held back.
I'm not sure what you're implying. Could you elaborate?

The search side of the API is currently being developed in Lucene-1458. 
1458 will not make it into 2.9. Therefore I agree that it is not very 
advantageous to switch to the new API right now for Lucene users. On the 
other hand, I don't think it hurts either.
> I personally would vote for reverting until a complete patch that 
> addresses both sides of the problem is submitted and a better solution 
> to cloning is put forth.
If we revert now and put a new flexible API like this into 3.x, which I 
think is necessary to utilize flexible indexing, then we'll have to wait 
until 4.0 before we can remove the old API. Disadvantages like the one 
you mentioned above, will then probably be present much longer.

I mentioned in the following thread that I have started working on a 
better way of cloning, which will actually be faster compared to the old 
API. I'll try to get the code out asap.

I'd be happy to discuss other API proposals that anybody brings up here, 
that have the same advantages and are more intuitive. We could also beef 
up the documentation and give a better example about how to convert a 
stream/filter from the old to the new API; a constructive suggestion 
that Uwe made at the ApacheCon.


> -Grant
> [1],
> [2] 

> and the rest of the discussion on that thread.
> [3] 

> On Jun 13, 2009, at 10:32 PM, Mark Miller wrote:
>> What was the big improvement with it again? Advanced, expert custom 
>> indexing chains require less casting or something right?

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message