lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Giffin <da...@giffin.org>
Subject Re: Token filter on multivalue field
Date Sat, 06 Jun 2009 17:38:31 GMT
I'm doing a combination of update processor and token filter. The
token filter is necessary to reduce the duplicates after stemming has
occurred.

David

2009/6/4 Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@corp.aol.com>:
> isn't better to use an UpdateProcessor  for this?
>
> On Thu, Jun 4, 2009 at 1:52 AM, Otis Gospodnetic
> <otis_gospodnetic@yahoo.com> wrote:
>>
>> Hello,
>>
>> It's ugly, but the first thing that came to mind was ThreadLocal.
>>
>>  Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>
>>
>>
>> ----- Original Message ----
>>> From: David Giffin <david@giffin.org>
>>> To: solr-user@lucene.apache.org
>>> Sent: Wednesday, June 3, 2009 1:57:42 PM
>>> Subject: Token filter on multivalue field
>>>
>>> Hi There,
>>>
>>> I'm working on a unique token filter, to eliminate duplicates on a
>>> multivalue field. My filter works properly for a single value field.
>>> It seems that a new TokenFilter is created for each value in the
>>> multivalue field. I need to maintain an array of used tokens across
>>> all of the values in the multivalue field. Is there a good way to do
>>> this? Here is my current code:
>>>
>>> public class UniqueTokenFilter extends TokenFilter {
>>>
>>>     private ArrayList words;
>>>     public UniqueTokenFilter(TokenStream input) {
>>>         super(input);
>>>         this.words = new ArrayList();
>>>     }
>>>
>>>     @Override
>>>     public final Token next(Token in) throws IOException {
>>>         for (Token token=input.next(in); token!=null; token=input.next())
{
>>>             if ( !words.contains(token.term()) ) {
>>>                 words.add(token.term());
>>>                 return token;
>>>             }
>>>         }
>>>         return null;
>>>     }
>>> }
>>>
>>> Thanks,
>>> David
>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

Mime
View raw message