lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Stripping Punctuation in a fieldType
Date Fri, 15 Jan 2010 18:42:21 GMT
If you haven't seen it, this page is invaluable for this kind of question:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.LetterTokenizerFactory
<http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.LetterTokenizerFactory>

LetterTokenizerFactory may well be your friend here, followed by
LowerCaserFilterFactory. There is a problem that it would
split "nation's" up into "nation" and "s", so searching on "nations"
wouldn't get a hit.

But you have equally ugly stuff with WhiteSpaceTokenizerFactory
as you're finding out.

It's actually quite rare for simple tokenizers like these to be
satisfactory
unless it's a field you can guarantee is indexed/searched in a very
controlled manner, say part numbers or words from a list. In your
example above, none of the three variants would get a hit if the
user searched for "nation". Is that what you want?

But no, Standard* don't have any stemming built in. And
what do you mean by "language specific functionality"?
They do NOT fold accents for instance if that's what
you're getting at.

Could you explain a bit about *why* you want this behavior?

HTH
Erick

On Fri, Jan 15, 2010 at 1:17 PM, David Seltzer <dseltzer@tveyes.com> wrote:

> I'm hesitant to change Tokenizers at the moment because what we have is
> working so nicely - or so I thought.
>
> What I'm looking for is case-insensitive search for words and numbers
> without any of the stemming features turned on. The new requirement is
> that we take punctuation out of the mix.
>
> Right now when I search for "Obama" I'm not getting any hits on "Obama."
>
> So I'm basically looking to strip punctuation. The consequence would be
> that "nation's", "nations" and "nations," would all be represented the
> same way.
>
> Would the StandardTokenizerFactory accomplish this?
> Does it have any language specific functionality?
> Does it do anything with stemming?
>
> Thanks for everyone's input!
>
> -Dave
>
>
>
> -----Original Message-----
> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
> Sent: Friday, January 15, 2010 12:42 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Stripping Punctuation in a fieldType
>
> > I'm trying to find the best way to set up a fieldType that
> > strips punctuation.
>
> Use solr.StandardTokenizerFactory that strips punctuations.
>
> Or if you do not care about alphanumeric or numeric queries use
> solr.LowerCaseTokenizerFactory that uses LetterTokenizer.
>
> I think the right way to do this is using a
> > CharacterFilter
> > of some type, but I can't seem to find any examples of how
> > to set this
> > up in a schema.xml file.
>
> If you want to use solr.MappingCharFilterFactory you need to write all
> punctiation characters to a text file manually. e.g. "," => ""
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message