lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sykes, Derek" <>
Subject RE: determining the type of a term - retrieving a payload
Date Thu, 14 Oct 2010 15:56:56 GMT
Hey Grant,

Fair point on the next(). In this case I'm iterating through the terms returned from a PrefixTermEnum
so I know they're in the index.

The analyser I'm using looks like this:

public class TypeSavingAnalyzer extends StandardAnalyzer {

    public TypeSavingAnalyzer(Version version) {

    public TokenStream tokenStream(String fieldName, Reader reader) {
        TokenStream result = super.tokenStream(fieldName, reader);
        return new TypeAsPayloadTokenFilter(result);


I'm creating the field with Field.TermVector.WITH_POSITIONS and see the little "p" indicator
in luke however see no indication of a payload (though am not 100% sure where to look) being
present and I still get false from isPayloadAvailable.

Basically what I'm trying to do is store in the payload some meta data so I know what type
the term is as at the moment when I get a list of terms back from the reader some are numeric.
 It seems NumericField doesn't allow payloads so it seems like I may have to find an alternative
method.  Any suggestions would be greatly appreciated!


-----Original Message-----
From: Grant Ingersoll [] 
Sent: 14 October 2010 15:22
Subject: Re: determining the type of a term - retrieving a payload

On Oct 13, 2010, at 11:37 AM, Sykes, Derek wrote:

> Hi there,
> I'm currently trying to work out how I can determine the type (string/number/date/etc)of
a term.  I've not seen any off the shelf way to do it so am trying to store a payload against
each term that records the type.
> I'm having a little trouble retrieving a payload I'd stored onto the term.  At the moment
I'm using the TypeAsPayloadTokenFilter, however I'll change this soon as it's not really what
I'm after.
> I'm then getting a TermEnum from either reader.terms() or using a PrefixTermEnum.
> For each term I want the payload from I'm running this function:
>    private String getPayload(Term term) {
>        String payload = null;
>        try {
>            TermPositions termPositions = reader.termPositions(term);
>  ;

 next() returns a boolean as to whether there is a valid entry, so you need to check that.
 You may not actually have a match for that term.

>            if (termPositions.isPayloadAvailable()) {
>                byte[] payloadBytes = new byte[termPositions.getPayloadLength()];
>                payloadBytes = termPositions.getPayload(payloadBytes, 0);
>                payload = new String(payloadBytes);
>                LOG.debug(payload);
>            }
>        }
>        catch( IOException e) {
> ...

What does your Analysis process look like?  Many of Lucene's analysis pieces don't bother
setting type.  Have you looked at the index with Luke?  That should show you the payloads.
 Also, have a look at the SpanTermQuery.  You can use the Spans object to step directly through
the position matches and get the payloads, if they are available.

Grant Ingersoll

To unsubscribe, e-mail:
For additional commands, e-mail:

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained
herein are not intended to be, and do not constitute, advice within the meaning of Section
975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received
this communication in error, please destroy all electronic and paper copies and notify the
sender immediately. Mistransmission is not intended to waive confidentiality or privilege.
Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor
electronic communications. This message is subject to terms available at the following link: If you cannot access these links, please notify
us by reply message and we will send the contents to you. By messaging with Morgan Stanley
you consent to the foregoing.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message