lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Seltzer" <dselt...@TVEyes.com>
Subject RE: Stripping Punctuation in a fieldType
Date Fri, 15 Jan 2010 18:17:32 GMT
I'm hesitant to change Tokenizers at the moment because what we have is
working so nicely - or so I thought.

What I'm looking for is case-insensitive search for words and numbers
without any of the stemming features turned on. The new requirement is
that we take punctuation out of the mix. 

Right now when I search for "Obama" I'm not getting any hits on "Obama."

So I'm basically looking to strip punctuation. The consequence would be
that "nation's", "nations" and "nations," would all be represented the
same way. 

Would the StandardTokenizerFactory accomplish this? 
Does it have any language specific functionality? 
Does it do anything with stemming?

Thanks for everyone's input!

-Dave



-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com] 
Sent: Friday, January 15, 2010 12:42 PM
To: solr-user@lucene.apache.org
Subject: Re: Stripping Punctuation in a fieldType

> I'm trying to find the best way to set up a fieldType that
> strips punctuation. 

Use solr.StandardTokenizerFactory that strips punctuations. 

Or if you do not care about alphanumeric or numeric queries use 
solr.LowerCaseTokenizerFactory that uses LetterTokenizer.

I think the right way to do this is using a
> CharacterFilter
> of some type, but I can't seem to find any examples of how
> to set this
> up in a schema.xml file. 

If you want to use solr.MappingCharFilterFactory you need to write all
punctiation characters to a text file manually. e.g. "," => ""


      

Mime
View raw message