lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Gindin <>
Subject Postings.getPayload() returns null
Date Fri, 23 Mar 2018 08:15:14 GMT
Hi all.

I have a simplified test, that defines an index with one document and one
field in it. Just one filter is defined in indexing analyzer that writes 1
byte to payload. My *goal *is to write numbers to payload for each position
of each term, that will be used in custom scoring formula implemented in
custom Query/Scorer.

When I'm trying to read written payload (in the custom query) I got NULL. I
suppose I should call posting.nextPosition() before calling getPayload()
method, but when I called nextPosition() I got an error:

java.lang.AssertionError: got line=field model

at __randomizedtesting.SeedInfo.seed([D334C9D1B5C155E3:2AAE4BE5481F4C8F]:0)

I also used SimpleTextCodec as you see to make sure that payload was really
written to index along with positions. It is really written. I probably do
something wrong with positions or reading it incorrectly or missed
something important.

*Question*: What am I doing wrong? How to read/write payload correctly?

Here is my test:

public class PayloadTest extends LuceneTestCase {
    private IndexSearcher searcher;
    private IndexReader reader;
    private byte[] payloadField = new byte[]{1};
    protected Directory directory;

    private class PayloadAnalyzer extends Analyzer {
        public TokenStreamComponents createComponents(String fieldName) {
            Tokenizer tokenizer = new LowerCaseTokenizer();
            PayloadFilter filter = new PayloadFilter(tokenizer, fieldName);
            return new TokenStreamComponents(tokenizer, filter);

    private class PayloadFilter extends TokenFilter {
        PayloadAttribute payloadAtt;
        PositionIncrementAttribute positionAtt;

        public PayloadFilter(TokenStream input, String fieldName) {
            payloadAtt = addAttribute(PayloadAttribute.class);
            positionAtt =
addAttribute(PositionIncrementAttribute.class); // I tried also
without position attribute here with the same error

        public boolean incrementToken() throws IOException {
            boolean hasNext = input.incrementToken();
            if (hasNext) {
                payloadAtt.setPayload(new BytesRef(payloadField));
                positionAtt.setPositionIncrement(1);  // I tried also
without position attribute here with the same error
                return true;
            } else {
                return false;

    public void setUp() throws Exception {
        directory = newDirectory();
        RandomIndexWriter writer = new RandomIndexWriter(random(), directory,
                newIndexWriterConfig(new PayloadAnalyzer())
                        .setCodec(new SimpleTextCodec()));

        Document doc = new Document();
        doc.add(new TextField("model", "ford focus", Field.Store.YES));

        reader = writer.getReader();

        searcher = newSearcher(reader);

    public void tearDown() throws Exception {

    public void testQuery() throws IOException {
        int limit = 20;
        try (IndexReader reader = {
            Query query = new CustomPhraseQuery(
                    Arrays.asList("ford", "focus"),
                    new HashMap<String, Float>() {{
                        put("model", 5.0f);
                    new HashMap<String, List<String>>() {{
                        put("ford", Arrays.asList("ford^1.0"));
                        put("focus", Arrays.asList("focus^1.0"));

            printSearchResults(limit, query, reader);

    private static void printSearchResults(final int limit, final Query query,
                                          final IndexReader reader)
throws IOException {
        IndexSearcher searcher = new IndexSearcher(reader);
        TopDocs docs =, limit);

        System.out.println(docs.totalHits + " found for query: " + query);

        for (final ScoreDoc scoreDoc : docs.scoreDocs) {

Here is the code from CustomPhraseQuery.scorer():

for (String field: fieldScores.keySet()) {
    final Terms fieldTerms = reader.terms(field);
    if (fieldTerms == null) {

    if (!fieldTerms.hasPositions())
        throw new IllegalStateException("Index does not contain positions");

    if (!fieldTerms.hasPayloads())
        throw new IllegalStateException("Index does not contain payloads");

    final TermsEnum te = fieldTerms.iterator();
    for (int j = 0; j < terms.length; j++) {
        final Term t = terms[j];

        if (t.field().equals(field) && te.seekExact(t.bytes())) {
            PostingsEnum postingsEnum = te.postings(null, PostingsEnum.ALL);

            int pos = postingsEnum.nextPosition();
            BytesRef payload = postingsEnum.getPayload();
            // assert payload.bytesEquals(new BytesRef(new byte[]{1}));

            // TODO: use payload in scoring formula
            fldScorers.add(new ConstTermScorer(this, t,
                    fieldScores.get(field) * termScores.get(t.text()),

Vadim Gindin

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message