lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ronnie Kolehmainen (JIRA)" <>
Subject [jira] Updated: (LUCENE-644) Contrib: another highlighter approach
Date Thu, 03 Aug 2006 17:44:15 GMT
     [ ]

Ronnie Kolehmainen updated LUCENE-644:


Mark, I played around a bit with the code in TokenSources and added a method which takes an
optional Query object. It will return a TokenSources.BestFragmentsTokenStream (which could
later be identified by Highlighter if needed) when the query is not null and term positions
are available. This token stream only holds the highlighted tokens and their surroundings.

These changes shave off about 50% of the time but is still two times slower than my first
example. Also, the fragments don't look as expected. The terms are highlighted but the surrounding
tokens are most often missing. I'm not sure why as I haven't dug deeper in the HighLighter
code. The tokens returned by TokenSources look fine though, with positions and in order...

It would certainly be nice if most changes could be made in TokenSources. This would allow
HighLigheter to be flexible as it is today, with Scorers and Formatters.

I won't have time to look at it anymore, at least for a while, but I'm posting my version
of TokenSources and a diff against current trunk here in case you want to have a peek at it

> Contrib: another highlighter approach
> -------------------------------------
>                 Key: LUCENE-644
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Other
>            Reporter: Ronnie Kolehmainen
>            Priority: Minor
>         Attachments:,, svn-diff.patch,,
> Mark Harwoods highlighter package is a great contribution to Lucene, I've used it a lot!
However, when you have *large* documents (fields), highlighting can be quite time consuming
if you increase the number of bytes to analyze with setMaxDocBytesToAnalyze(int). The default
value of 50k is often too low for indexed PDFs etcetera, which results in empty highlight
> This is an alternative approach using term position vectors only to build fragment info
objects. Then a StringReader can read the relevant fragments and skip() between them. This
is a lot faster. Also, this method uses the *entire* field for finding the best fragments
so you're always guaranteed to get a highlight snippet.
> Because this method only works with fields which have term positions stored one can check
if this method works for a particular field using following code (taken from
>         TermFreqVector tfv = (TermFreqVector) reader.getTermFreqVector(docId, field);
>         if (tfv != null && tfv instanceof TermPositionVector)
>         {
>           // use FulltextHighlighter
>         }
>         else
>         {
>           // use standard Highlighter
>         }
> Someone else might find this useful so I'm posting the code here.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message