lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolás Lichtmaier <nicol...@wolfram.com.INVALID>
Subject Help with token streams and graphs
Date Wed, 20 Mar 2019 17:52:33 GMT
I'm trying to make synonyms work right and for that I'm trying to 
understand better graphs in a token stream.

For that purpose I've built this code:

             Builder builder = CustomAnalyzer.builder();             
builder.withTokenizer(StandardTokenizerFactory.class);             
MySynonymGraphFilterFactory.registerSynonyms(Arrays.asList(             
         Arrays.asList("go to", "navigate", "open")                    

));             
builder.addTokenFilter(*MySynonymGraphFilterFactory*.class, "synonyms", 
"unused");

MySynonymGraphFilterFactory is just a hack to pass a list of lists for 
synonyms. It expands everything mapping everything to everything.

             builder.addTokenFilter(*FlattenGraphFilterFactory*.class); 
/// nothing changes with this!///            Analyzer analyzer = 
builder.build();             TokenStream ts = analyzer.tokenStream("*", 
new StringReader("go to the webpage!"));

Then I call a function that just dumps terms, position increments and 
position lengths:

             System.out.println(LoggingFilter.tokenStreamToString(ts));

What I don't understand is this. I get the same output whether I include 
FlattenGraphFilter or not. This is the output:

    navigate<2> (0)open<2> (0)go  to  the  webpage

(angle brackets show position lengths of the preceding term; parenthesis 
show position increments of the following term)

There's something I'm not understanding here. I'd thought that 
flattening the stream meant that no token will have position length > 
1... was I wrong? I would greatly appreciate any help with understanding 
this.

Thanks!

Nicolás.-


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message