lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Della Bitta <michael.della.bi...@appinions.com>
Subject Re: Applying Tokenizers and Filters to CopyFields
Date Wed, 25 Mar 2015 20:41:17 GMT
Two other things I noticed:

1. You probably don't want to store your copyFields. That's literally going
to be the same information each time.

2. Your expectation "the pre-processed version of the text is added to the
index" may be incorrect. Anything done in <analyzer type="query"> sections
actually happens at query time. Not sure if that's significant for you.


Michael Della Bitta

Senior Software Engineer

o: +1 646 532 3062

appinions inc.

“The Science of Influence Marketing”

18 East 41st Street

New York, NY 10017

t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>

On Wed, Mar 25, 2015 at 4:27 PM, Ahmet Arslan <iorixxx@yahoo.com.invalid>
wrote:

> Hi Martin,
>
> fq means filter query. May be you want to use qf (query fields) parameter
> of edismax?
>
>
>
> On Wednesday, March 25, 2015 9:23 PM, Martin Wunderlich <martin_wu@gmx.net>
> wrote:
> Hi all,
>
> I am wondering what the process is for applying Tokenizers and Filter (as
> defined in the FieldType definition) to field contents that result from
> CopyFields. To be more specific, in my Solr instance, Iwould like to
> support query expansion by two means: removing stop words and adding
> inflected word forms as synonyms.
>
> To use a specific example, let’s say I have the following sentence to be
> indexed (from a Wittgenstein manuscript):
>
> "Was zum Wesen der Welt gehört, kann die Sprache nicht ausdrücken.“
>
>
> This sentence will be indexed in a field called „original“ that is defined
> as follows:
>
> <field name="original" type="text_original" indexed="true" stored="true"
> required="true“/>
>
>     <fieldType name="text_windex_original" class="solr.TextField"
> positionIncrementGap="100">
>       <analyzer type="index">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>       </analyzer>
>     </fieldType>
>
>
> Then, in order to create fields for the two types of query expansion, I
> have set up specific fields for this:
>
> - one field where stopwords are removed both on the indexed content and
> the query. So, if the users is searching for a phrase like „der Sprache“,
> Solr should still find the segment above, because the determiners („der“
> and „die“) are removed prior to indexing and prior to querying,
> respectively. This field is defined as follows:
>
> <field name="stopwords_removed" type="text_stopwords_removed"
> indexed="true" stored="true" required="true“/>
>
>     <fieldType name="text_stopwords_removed" class="solr.TextField"
> positionIncrementGap="100">
>       <analyzer type="index">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words=„stopwords_de.txt" format="snowball"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords_de.txt" format="snowball"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>     </fieldType>
>
>
> - a second field where synonyms are added to the query so that more
> segments will be found. For instance, if the user is searching for the
> plural form „Sprachen“, Solr should return the segment above, due to this
> entry in the synonyms file: "Sprache,Sprach,Sprachen“. This field is
> defined as follows:
>
> <field name="expanded" type="text_multiplied" indexed="true" stored="true"
> required="true“/>expanded
>
>     <fieldType name="text_expanded" class="solr.TextField"
> positionIncrementGap="100">
>       <analyzer type="index">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords_de.txt" format="snowball"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class="solr.StandardTokenizerFactory"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords_de.txt" format="snowball"/>
>         <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms_de.txt" ignoreCase="true" expand="true"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>       </analyzer>
>     </fieldType>
>
> Finally, to avoid having to specify three fields with identical content in
> the import documents, I am defining the two fields for query expansion as
> copyFields:
>
>   <copyField source="original" dest="stopwords_removed"/>
>   <copyField source="original" dest="expanded“/>
>
> Now, my expectation would be as follows:
> - during import, two temporary fields are created by copying content from
> the original field
> - these two temporary fields are then pre-processed as per the definitions
> above
> - the pre-processed version of the text is added to the index
> - then, the user can search for „Sprache“, „sprache“, „Sprachen“ or „der
> Sprache“ and will always get the segment above as a matching result.
>
> However, what happens actually is that I get matches only for „Sprache“
> and „sprache“.
>
> The other thing that strikes as odd, is that when I restrict the search to
> one of the fields only using the „fq“ parameter, I get no results. For
> instance:
>
> http://localhost:8983/solr/windex/select?q=Sprache&fq=original&wt=json&indent=true
> <
> http://localhost:8983/solr/windex/select?q=Sprache&fq=original&wt=json&indent=true
> >
>
> will return no matches. I would expected that using the fq parameter the
> user can specify what type of search (s)he would like to carry out: A
> standard search (field original) or an expanded search (one of the other
> two fields).
>
> For debugging, I have checked the analysis and results seem ok (posted
> below).
> Apologies for the long post, but I am really a bit stuck here (even after
> doing a lot of reading and googling). It is probably something simple that
> I missing.
> Thanks a lot in advance for any help.
>
> Cheers,
>
> Martin
>
>
> ST
> Was
> zum
> Wesen
>
> der
> Welt
> gehört
> kann
> die
> Sprache
> nicht
> ausdrücken
> SF
> Was
> zum
> Wesen
>
> Welt
> gehört
> kann
> die
> Sprache
> nicht
> ausdrücken
> LCF
> was
> zum
> wesen
>
> welt
> gehört
> kann
> die
> sprache
> nicht
> ausdrücken
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message