lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Sekiguchi <koji.sekigu...@rondhuit.com>
Subject Re: Issues trying to boost phrase containing stop word
Date Thu, 20 Jul 2017 01:47:57 GMT
Hi Shamik,

How about using ShingleFilter which constructs token n-grams from a token stream?

http://lucene.apache.org/core/6_6_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilter.html

As for "about dynamic block", ShingleFilter produces "about dynamic" and "dynamic block".

Thanks,

Koji

On 2017/07/20 5:54, Shamik Bandopadhyay wrote:
> Hi,
> 
>    I'm trying to show titles with exact query phrase match at the top of the
> result. That includes supporting stop words as part of the phrase. For e.g.
> if I'm using "about dynamic "block" , I expect the title with "About
> Dynamic Blocks" to appear at the top. Since the title field uses
> stoprword filter factory as part of its analysis chain, I decided to create
> a copyfield of title and use that in search with a higher boost. That
> didn't seem to work either. Although it brought back the expected document
> at the top, it excluded documents with title "Dynamic Block Grip
> Reference", to be precise content which doesn't have "about" in title or
> subject. Even setting the default operator to OR didn't make any
> difference. Here's the entry from config.
> 
> <field name="title" type="adsktext" indexed="true" stored="true"
> multiValued="true"/>
> <field name="titleExact" type="textExact" indexed="true" stored="true"  />
> <field name="subject" type="adsktext" indexed="true" stored="true"/>
> <field name="description" type="adsktext" indexed="true" stored="true"/>
> <copyField source="title" dest="titleExact" />
> 
> 
> <fieldType name="adsktext" class="solr.TextField"
> positionIncrementGap="100" autoGeneratePhraseQueries="false">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.SynonymGraphFilterFactory"
> synonyms="synonyms/synonyms.txt" ignoreCase="true" expand="true"/>
> <!--<filter class="solr.KeywordRepeatFilterFactory"/>-->
> <!--<filter class="solr.CommonGramsFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>-->
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="0" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="0"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
> <!--<filter class="solr.PorterStemFilterFactory"/>-->
> <filter class="com.nicholasding.search.solr.LemmatizerFilterFactory"/>
> <!--<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>-->
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.SynonymGraphFilterFactory"
> synonyms="synonyms/synonyms.txt" ignoreCase="true" expand="true"/>
> <!--<filter class="solr.KeywordRepeatFilterFactory"/>-->
> <!--<filter class="solr.CommonGramsFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>-->
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="0" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="0"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" />
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
> <!--<filter class="solr.PorterStemFilterFactory"/>-->
> <filter class="com.nicholasding.search.solr.LemmatizerFilterFactory"/>
> <!--<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>-->
> </analyzer>
> </fieldType>
> 
> <fieldType name="textExact" class="solr.TextField"
> positionIncrementGap="100" >
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LimitTokenCountFilterFactory" maxTokenCount="20"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LimitTokenCountFilterFactory" maxTokenCount="20"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> 
> Request handler:
> <requestHandler name="/browse" class="solr.SearchHandler">
>       <lst name="defaults">
>         <str name="echoParams">explicit</str>
>         <str name="wt">velocity</str>
>         <str name="v.template">browse</str>
>         <str name="v.layout">layout</str>
>         <str name="title">Solritas</str>
>         <str name="q.op">AND</str>
>          <str name="defType">edismax</str>
>         <str name="qf">title^5 titleExact^15 subject^3 description^2</str>
> <!--<str name="pf">title^10 titleExact^100</str>-->
>         <str name="mm">100%</str>
>         <str name="q.alt">*:*</str>
>         <str name="rows">10</str>
>         <str name="fl">*,score</str>
>       </lst>
>    </requestHandler>
> 
> Sample data:
> <add>
> <doc>
> <field name="id">SOLR1000</field>
> <field name="title">About Dynamic Blocks</field>
> <field name="subject">Dynamic blocks contain rules, or parameters, for how
> to change the appearance of the block reference when it is inserted in the
> drawing. With dynamic blocks you can insert one block that can change
> shape, size, or configuration instead of inserting one of many static block
> definitions. For example, instead of creating multiple interior door blocks
> of different sizes, you can create one resizable door block. You author
> dynamic blocks with either constraint parameters or action parameters.
> Note: Using both constraint parameters and action parameters in the same
> block definition is not recommended. Add Constraints In a block definition,
> constraint parameters Associate objects with one another Restrict geometry
> or dimensions The following example shows a block reference with a
> constraint (in gray) and a constraint parameter (blue with grip). Once the
> block is inserted into the drawing, the constraint parameters can be edited
> as properties by using the Properties palette. Add Actions and Parameters
> In a block definition, actions and parameters provide rules for the
> behavior of a block once it is inserted into the drawing. Depending on the
> specified block geometry or parameter, you can associate an action to that
> parameter. The parameter is represented as a grip in the drawing. When you
> edit the grip, the associated action determines what will change in the
> block reference. Like constraint parameters, action parameters can be
> changed using the Properties palette.</field>
> <field name="description">Dynamic blocks contain rules, or parameters, for
> how to change the appearance of the block reference when it is inserted in
> the drawing.</field>
> </doc>
> <doc>
> <field name="id">SOLR1001</field>
> <field name="title">About Creating Dynamic Blocks</field>
> <field name="subject">This table gives an overview of the steps required
> add behaviors that make blocks dynamic. Plan the block content. Know how
> the block should change or move, and what parts will depend on the others.
> Example: The block will be resizable, and after it is resized, additional
> geometry is displayed. Draw the geometry. Draw the block geometry in the
> drawing area or the Block Editor. Note: If you will use visibility states
> to change how geometry is displayed, you may not want to include all the
> geometry at this point. Add parameters. Add either individual parameters or
> parameter sets to define geometry that will be affected by an action or
> manipulation. Keep in mind the objects that will be dependent on one
> another. Add actions. If you are working with action parameters, if
> necessary, add actions to define what will happen to the geometry when it
> is manipulated. Define custom properties. Add properties that determine how
> the block is displayed in the drawing area. Custom properties affect grips,
> labels, and preset values for block geometry. Test the block. On the
> ribbon, in the Block Editor contextual tab, Open/Save panel, click Test
> Block to test the block before you save it.</field>
> <field name="description">This table gives an overview of the steps
> required add behaviors that make blocks dynamic.</field>
> </doc>
> <doc>
> <field name="id">SOLR1002</field>
> <field name="title">About Modifying Dynamic Block Definitions</field>
> <field name="subject">Use the Block Editor to edit, correct, and save a
> block definition. Correct Errors in Action Parameters A yellow alert icon (
> ) is displayed when A parameter is not associated with an action An action
> is not associated with a parameter or selection set To correct these
> errors, hover over the yellow alert icon until the tooltip displays a
> description of the problem. Then double-click the constraint and follow the
> prompts. Save Dynamic Blocks When you save a block definition, the current
> values of the geometry and parameters in the block become the default
> values for the block reference. The default visibility state for the block
> reference is the visibility state at the top of the list in the Manage
> Visibility States dialog box. Note: If you click File menu Save while you
> are in the Block Editor, you will save the drawing but not the block
> definition. You must specifically save the block definition while you are
> in the Block Editor.</field>
> <field name="description">Use the Block Editor to edit, correct, and save a
> block definition.</field>
> </doc>
> <doc>
> <field name="id">SOLR1003</field>
> <field name="title">About Adding Parameters to Dynamic Blocks</field>
> <field name="subject">Parameters determine the geometry that will be
> affected by an action when you manipulate a block reference. When you add a
> parameter to a dynamic block definition, grips are displayed at key points
> of the parameter. Key points are the parts of a parameter that you use to
> manipulate the block reference. For example, a linear parameter has key
> points at its base point and end point. You can manipulate the parameter
> distance from either key point. You can specify grip size and color for
> display in the Block Editor. This setting does not affect the size and
> color of the grips in a block reference.</field>
> <field name="description">Parameters determine the geometry that will be
> affected by an action when you manipulate a block reference.</field>
> </doc>
> <doc>
> <field name="id">SOLR1004</field>
> <field name="title">About Adding Actions to Dynamic Blocks</field>
> <field name="subject">Actions define how the geometry of a dynamic block
> reference will move or change when its grips are manipulated. In general,
> you associate an action with a parameter and the following: Key point . The
> point on a parameter that drives the action. Selection set . The geometry
> that will be affected by the action. When you move the grip in the example
> above, only the geometry in the selection set is stretched. Specify
> Distance and Angle Override Values Distance multiplier and angle offset
> override properties allow you to specify a factor by which a parameter
> value is increased or decreased. Action overrides are properties of actions
> that have no effect on the block reference until it is manipulated in a
> drawing. Use distance multiplier overrides with the following actions: Move
> action Stretch action Polar Stretch action You can specify these action
> override properties on the command line when you add an action to a dynamic
> block definition. You can also specify these properties in the Properties
> palette when you select an action in the Block Editor.</field>
> <field name="description">Actions define how the geometry of a dynamic
> block reference will move or change when its grips are manipulated.</field>
> </doc>
> <doc>
> <field name="id">SOLR1005</field>
> <field name="title">Dynamic Block Grip Reference</field>
> <field name="subject">This table describes the grips and how they're used.
> Grip Type Grip Movement or Result Parameters: Associated Actions Standard
> Within a plane in any direction Base: None Point: Move, Stretch, Polar:
> Move, Scale, Stretch, Polar Stretch, Array XY: Move, Scale, Stretch, Array
> Linear Back and forth in a defined direction or along an axis Linear: Move,
> Scale, Stretch, Array Rotation Around an axis Rotation: Rotate Flip
> Switches to a mirror image of the block geometry Flip: Flip Alignment
> Within a plane in any direction; when moved over an object, triggers the
> block reference to align with the object Alignment: None (action is
> implied) Lookup Displays a list of values Visibility: None (action is
> implied) Lookup: Lookup</field>
> <field name="description">This table describes the grips and how they're
> used.</field>
> </doc>
> <doc>
> <field name="id">SOLR1006</field>
> <field name="title">To Open a Drawing Saved as a Dynamic Block (Block
> Editor)</field>
> <field name="subject">Click the Application button Open Drawing. Open the
> drawing file that is saved as a block. An alert states that the drawing
> contains authoring elements. In the alert dialog box, click Yes to open the
> drawing in the Block Editor.</field>
> <field name="description">None</field>
> </doc>
> </add>
> 
> 
> Here's my query:
> http://localhost:8983/solr/techproducts/browse?q=About%20dynamic%20blocks&wt=xml&fl=title,subject,score&facet=off&hl=off&spellcheck=off&qf=title
> ^5%20titleExact^1%20subject^1%20description^1&debugQuery=true&rows=100&q.op=OR
> 
> It skips SOLR1004 and SOLR1005 since they don't have the term "about".
> Adding q.op=OR didn't make any difference in result. Here's the debug query
> for both q.op=AND and OR
> 
> AND:
> <str name="parsedquery">(+(+DisjunctionMaxQuery((titleExact:about))
> +DisjunctionMaxQuery((titleExact:dynamic | description:dynamic |
> (title:dynamic)^5.0 | subject:dynamic))
> +DisjunctionMaxQuery((titleExact:blocks | description:block |
> (title:block)^5.0 | subject:block))))/no_coord</str>
> 
> <str name="parsedquery_toString">+(+(titleExact:about) +(titleExact:dynamic
> | description:dynamic | (title:dynamic)^5.0 | subject:dynamic)
> +(titleExact:blocks | description:block | (title:block)^5.0 |
> subject:block))</str>
> 
> OR:
> <str name="parsedquery">(+(DisjunctionMaxQuery((titleExact:about))
> DisjunctionMaxQuery((titleExact:dynamic | description:dynamic |
> (title:dynamic)^5.0 | subject:dynamic))
> DisjunctionMaxQuery((titleExact:blocks | description:block |
> (title:block)^5.0 | subject:block)))~3)/no_coord</str>
> 
> <str name="parsedquery_toString">+(((titleExact:about) (titleExact:dynamic
> | description:dynamic | (title:dynamic)^5.0 | subject:dynamic)
> (titleExact:blocks | description:block | (title:block)^5.0 |
> subject:block))~3)</str>
> 
> Apologies for the long thread, but not sure what I'm doing wrong. I'll
> appreciate if you someone can provide pointers. If there's a different
> approach to solving this issue, please let me know.
> 
> Thanks,
> Shamik
> 

Mime
View raw message