lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christoph Goller (JIRA)" <>
Subject [jira] [Commented] (LUCENE-8000) Document Length Normalization in BM25Similarity correct?
Date Fri, 20 Oct 2017 08:35:00 GMT


Christoph Goller commented on LUCENE-8000:

??My point is that defaults are for typical use-cases, and the default of discountOverlaps
meets that goal. It results in better (measured) performance for many tokenfilters that are
commonly used such as common-grams, WDF, synonyms, etc. I ran these tests before proposing
the default, it was not done flying blind.??

Understood.* I have not experienced any problems with the current default* and I have the
option to set discountOverlaps to false. Therefore it's ok for me if the ticket gets closed.

I only think about this out of "scientific" curiosity in the context of  relevance tuning.

What benchmarks have you used for measuring performance?

Is your opinion based on tests with Lucene Classic Similarity (it also uses discountOverlaps
= true) or also on tests with BM25.

Have you any idea / explaination why relevancy is better using discountOverlaps = true. My
naive guess would be that since stopwords or synonyms are either used on all documents or
on none and therefore it should not make much difference whether we count overlaps or not.
Is the explaination that for some documents many stopwords / synonyms / WDF splits are used
and for others not (for the same field).

Sorry for bothering you with these questions. It's only my curiosity and mayb Jira is nto
the right place for this.

> Document Length Normalization in BM25Similarity correct?
> --------------------------------------------------------
>                 Key: LUCENE-8000
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Christoph Goller
>            Priority: Minor
> Length of individual documents only counts the number of positions of a document since
discountOverlaps defaults to true.
> {code}
>  @Override
>   public final long computeNorm(FieldInvertState state) {
>     final int numTerms = discountOverlaps ? state.getLength() - state.getNumOverlap()
: state.getLength();
>     int indexCreatedVersionMajor = state.getIndexCreatedVersionMajor();
>     if (indexCreatedVersionMajor >= 7) {
>       return SmallFloat.intToByte4(numTerms);
>     } else {
>       return SmallFloat.floatToByte315((float) (1 / Math.sqrt(numTerms)));
>     }
>   }}
> {code}
> Measureing document length this way seems perfectly ok for me. What bothers me is that
> average document length is based on sumTotalTermFreq for a field. As far as I understand
that sums up totalTermFreqs for all terms of a field, therefore counting positions of terms
including those that overlap.
> {code}
>  protected float avgFieldLength(CollectionStatistics collectionStats) {
>     final long sumTotalTermFreq = collectionStats.sumTotalTermFreq();
>     if (sumTotalTermFreq <= 0) {
>       return 1f;       // field does not exist, or stat is unsupported
>     } else {
>       final long docCount = collectionStats.docCount() == -1 ? collectionStats.maxDoc()
: collectionStats.docCount();
>       return (float) (sumTotalTermFreq / (double) docCount);
>     }
>   }
> }
> {code}
> Are we comparing apples and oranges in the final scoring?
> I haven't run any benchmarks and I am not sure whether this has a serious effect. It
just means that documents that have synonyms or in my use case different normal forms of tokens
on the same position are shorter and therefore get higher scores  than they should and that
we do not use the whole spectrum of relative document lenght of BM25.
> I think for BM25  discountOverlaps  should default to false. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message