lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lajos <la...@protulae.com>
Subject Re: Help! Issue with tokens in custom synonym filter
Date Mon, 31 Aug 2009 17:22:00 GMT
Hi David & Ahmet,

I hadn't seen the SynonymTokenFilter from Lucene, so that helped. 
Ultimately, however, it seems I was pretty much doing the right thing, 
although my token type might have been wrong.

Unfortunately, while the tokens are being returned properly (AFAIK), 
when I do a query using one of the synonyms, I can't get any results. 
This is not the case if I just directly code in the synonym into the 
synonyms file with the standard solr synonym filter.

So I'll have to keep on hacking away ;)

Regarding generating the file from WordNet, we'd considered that but our 
requirements essentially mean we have to do the heavy lifting within the 
filter itself. Not that I'm opposed, it is just that I'm apparently 
missing something simple still.

Thanks for the replies.

Lajos


Smiley, David W. wrote:
> Although this is not a direct answer to your question, you may want to consider generating
a synonyms file from wordnet.  Then, you can use the standard synonym filter in Solr.  The
only downside to this is that the synonym file might be pretty large... but you've probably
got some large file for wordnet data any way.
> 
> ~ David Smiley
>  Author: http://www.packtpub.com/solr-1-4-enterprise-search-server
> 
> 
> 
> On 8/31/09 10:32 AM, "Lajos" <lajos@protulae.com> wrote:
> 
> Hi all,
> 
> I've been writing some custom synonym filters and have run into an issue
> with returning a list of tokens. I have a synonym filter that uses the
> WordNet database to extract synonyms. My problem is how to define the
> offsets and position increments in the new Tokens I'm returning.
> 
> For an input token, I get a list of synonyms from the WordNet database.
> I then create a List<Token> of those results. Each Token is created with
> the same startOffset, endOffset and positionIncrement of the input
> Token. Is this correct? My understanding from looking at the Lucene
> codebase is that the startOffset/endOffset should be the same, as we are
> referring to the same term in the original text. However, I don't quite
> get the positionIncrement. I understand that it is relative to the
> previous term ... does this mean all my synonyms should have a
> positionIncrement of 0? But whether I use 0 or the positionIncrement of
> the original input Token, Solr seems to ignore the returned tokens ...
> 
> This is a summary of what is in my filter:
> 
> *************************************************
> 
> private Iterator<Token> output;
> private ArrayList<Token> synonyms = null;
> 
> public Token next(Token in) throws IOException {
>    if (output != null) {
>      // Here we are just outputing matched synonyms
>      // that we previously created from the input token
>      // The input token has already been returned
>      if (output.hasNext()) {
>        return output.next();
>      } else {
>        return null;
>      }
>    }
> 
>    synonyms = new ArrayList<Token>();
> 
>    Token t = input.next(in);
>    if (t == null) return null;
> 
>    String value = new String(t.termBuffer(), 0,
>      t.termLength()).toLowerCase();
> 
>    // Get list of WordNet synonyms (code removed)
>    // Iterate thru WordNet synonyms
>    for (String wordNetSyn : wordNetSyns) {
>      Token synonym = new Token(t.startOffset(), t.endOffset(),
> t.type());          synonym.setPositionIncrement(t.getPositionIncrement());
>      synonym.setTermBuffer(wordNetSyn .toCharArray(), 0,
>        wordNetSyn .length());
>      synonyms.add(synonym);
>    }
> 
>    output = synonyms.iterator();
> 
>    // Return the original word, we want it
>    return t;
> }
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> 
> No virus found in this incoming message.
> Checked by AVG - www.avg.com 
> Version: 8.5.409 / Virus Database: 270.13.71/2334 - Release Date: 08/29/09 17:51:00
> 

Mime
View raw message