lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Smiley, David W." <dsmi...@mitre.org>
Subject Re: Help! Issue with tokens in custom synonym filter
Date Mon, 31 Aug 2009 15:10:00 GMT
Although this is not a direct answer to your question, you may want to consider generating
a synonyms file from wordnet.  Then, you can use the standard synonym filter in Solr.  The
only downside to this is that the synonym file might be pretty large... but you've probably
got some large file for wordnet data any way.

~ David Smiley
 Author: http://www.packtpub.com/solr-1-4-enterprise-search-server



On 8/31/09 10:32 AM, "Lajos" <lajos@protulae.com> wrote:

Hi all,

I've been writing some custom synonym filters and have run into an issue
with returning a list of tokens. I have a synonym filter that uses the
WordNet database to extract synonyms. My problem is how to define the
offsets and position increments in the new Tokens I'm returning.

For an input token, I get a list of synonyms from the WordNet database.
I then create a List<Token> of those results. Each Token is created with
the same startOffset, endOffset and positionIncrement of the input
Token. Is this correct? My understanding from looking at the Lucene
codebase is that the startOffset/endOffset should be the same, as we are
referring to the same term in the original text. However, I don't quite
get the positionIncrement. I understand that it is relative to the
previous term ... does this mean all my synonyms should have a
positionIncrement of 0? But whether I use 0 or the positionIncrement of
the original input Token, Solr seems to ignore the returned tokens ...

This is a summary of what is in my filter:

*************************************************

private Iterator<Token> output;
private ArrayList<Token> synonyms = null;

public Token next(Token in) throws IOException {
   if (output != null) {
     // Here we are just outputing matched synonyms
     // that we previously created from the input token
     // The input token has already been returned
     if (output.hasNext()) {
       return output.next();
     } else {
       return null;
     }
   }

   synonyms = new ArrayList<Token>();

   Token t = input.next(in);
   if (t == null) return null;

   String value = new String(t.termBuffer(), 0,
     t.termLength()).toLowerCase();

   // Get list of WordNet synonyms (code removed)
   // Iterate thru WordNet synonyms
   for (String wordNetSyn : wordNetSyns) {
     Token synonym = new Token(t.startOffset(), t.endOffset(),
t.type());          synonym.setPositionIncrement(t.getPositionIncrement());
     synonym.setTermBuffer(wordNetSyn .toCharArray(), 0,
       wordNetSyn .length());
     synonyms.add(synonym);
   }

   output = synonyms.iterator();

   // Return the original word, we want it
   return t;
}


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message