lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "DM Smith (JIRA)" <>
Subject [jira] Commented: (LUCENE-1333) Token implementation needs improvements
Date Mon, 04 Aug 2008 20:46:44 GMT


DM Smith commented on LUCENE-1333:

All the code has been migrated to use the reuse interface. I have run all the tests and they
pass. (There is a weird dependency. One test depends upon demo. I hacked that to work.) I
did not test to see if the code were any slower or faster. I'm not sure how one would do that.
I think it should be faster since it doesn't bounce back and forth with termText and termBuffer.

I did not improve the code to use char[] instead of String. The places that can be improved
call Token.setTermBuffer(String). I don't think these are necessary to this issue being resolved.

A couple of other minor opportunities for char[] manipulation via o.a.l.util.ArrayUtil.

Here and there, there is a need to remove leading and trailing whitespace from input before
(sub-)tokenizing it. Currently this is done with String.trim(), even when working with char[].
It is sub-optimal as marshalling it into a String involves allocation and copying. Likewise,
so does getting it out. It would be better to have int trim(char[]) which shifts leading spaces
off the front and returns the length of the string without trailing spaces.

There is a "randomize" routine that shuffles an array. While this is only used in one place,
it appears to be general purpose array manipulation.

> Token implementation needs improvements
> ---------------------------------------
>                 Key: LUCENE-1333
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>    Affects Versions: 2.3.1
>         Environment: All
>            Reporter: DM Smith
>            Priority: Minor
>             Fix For: 2.4
>         Attachments: LUCENE-1333-analysis.patch, LUCENE-1333-analyzers.patch, LUCENE-1333-core.patch,
LUCENE-1333-highlighter.patch, LUCENE-1333-instantiated.patch, LUCENE-1333-lucli.patch, LUCENE-1333-memory.patch,
LUCENE-1333-miscellaneous.patch, LUCENE-1333-queries.patch, LUCENE-1333-snowball.patch, LUCENE-1333-wikipedia.patch,
LUCENE-1333-wordnet.patch, LUCENE-1333-xml-query-parser.patch, LUCENE-1333.patch, LUCENE-1333.patch,
> This was discussed in the thread (not sure which place is best to reference so here are
> or to see it all at once:
> Issues:
> 1. JavaDoc is insufficient, leading one to read the code to figure out how to use the
> 2. Deprecations are incomplete. The constructors that take String as an argument and
the methods that take and/or return String should *all* be deprecated.
> 3. The allocation policy is too aggressive. With large tokens the resulting buffer can
be over-allocated. A less aggressive algorithm would be better. In the thread, the Python
example is good as it is computationally simple.
> 4. The parts of the code that currently use Token's deprecated methods can be upgraded
now rather than waiting for 3.0. As it stands, filter chains that alternate between char[]
and String are sub-optimal. Currently, it is used in core by Query classes. The rest are in
contrib, mostly in analyzers.
> 5. Some internal optimizations can be done with regard to char[] allocation.
> 6. TokenStream has next() and next(Token), next() should be deprecated, so that reuse
is maximized and descendant classes should be rewritten to over-ride next(Token)
> 7. Tokens are often stored as a String in a Term. It would be good to add constructors
that took a Token. This would simplify the use of the two together.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message