lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivana Spasojevic <ivanaspasojevic87...@gmail.com>
Subject DelimitedBoostTokenFilterFactory Issue - Boosting and BooleanSimilarity
Date Thu, 16 Apr 2020 12:29:02 GMT
Hi there,

I’m developing custom java application with lucene 8.5.0.

I've tried to use DelimitedBoostTokenFilterFactory but I have a problem, so
please help me if I'm doing something wrong.

When I’m using BM25Similarity and delimitedBoost filter everything works as
expected, but if I switch to BooleanSimilarity nothing happens.

Parsed query looks ok. It has synonyms with proper boost value, but the
final score hasn’t changed.

I’m using StandardAnalyzer for search, and my SynonymGraphFilter has
default configuration:

Map<String, String> synonymParam = new HashMap<>();
            synonymParam.put("synonyms", synonymFileName);
            synonymParam.put("ignoreCase", "true");
            synonymParam.put("format", "solr");
            synonymParam.put("expand","true");

synonymParam.put("tokenizerFactory","org.apache.lucene.analysis.core.WhitespaceTokenizerFactory");
Map<String, String> delimitedBoostTokenFilterMap = new HashMap<>();
delimitedBoostTokenFilterMap.put("delimiter", "|");
Analyzer customAnalyzer = CustomAnalyzer.builder(Paths.get(synonymFolder))
                    .withTokenizer(StandardTokenizerFactory.NAME)
                    .addTokenFilter(SynonymGraphFilterFactory.NAME,
synonymParam)
                    .addTokenFilter(DelimitedBoostTokenFilterFactory.NAME,
delimitedBoostTokenFilterMap)
                    .build();


Here’s my debug output and some additional info:

Query: +Synonym(morphology_term_original_name_key:neoplasm^0.7
morphology_term_original_name_key:tumor^0.8
morphology_term_original_name_key:tumour^0.6)

1.0 = weight(Synonym(morphology_term_original_name:neoplasm^0.7
morphology_term_original_name:tumor^0.8
morphology_term_original_name:tumour^0.6) in 0) [BooleanSimilarity], result
of:
1.0 = score(BooleanWeight), computed from:
1.0 = boost, query boost

If I use the BM25Similarity, the printout is as follows:

0.75188845 = weight(Synonym(morphology_term_original_name:neoplasm^0.7
morphology_term_original_name:tumor^0.8
morphology_term_original_name:tumour^0.6) in 0) [BM25Similarity], result of:
0.75188845 = score(freq=0.8), computed as boost * idf * tf from:
1.3862944 = idf, computed as log(1 + (N – n + 0.5) / (n + 0.5)) from:
1 = n, number of documents containing term
5 = N, total number of documents with field
0.5423729 = tf, computed as freq / (freq + k1 * (1 – b + b * dl / avgdl))
from:
0.8 = termFreq=0.8
1.2 = k1, term saturation parameter
0.75 = b, length normalization parameter
1.0 = dl, length of field
2.4 = avgdl, average length of field

Thanks in advance!

Ivana

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message