lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Rowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-7708) Track PositionLengthAttribute abuse
Date Fri, 24 Feb 2017 17:41:44 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883169#comment-15883169
] 

Steve Rowe commented on LUCENE-7708:
------------------------------------

+1 to the idea, but some tests are failing with the patch:

{noformat}
   [junit4] Tests with failures [seed: 4D8AED66905F8617]:
   [junit4]   - org.apache.lucene.analysis.shingle.ShingleFilterTest.testOutputUnigramsIfNoShinglesSingleTokenCase
   [junit4]   - org.apache.lucene.analysis.shingle.ShingleFilterTest.testOutputUnigramsIfNoShinglesWithMultipleInputTokens
   [junit4]   - org.apache.lucene.analysis.shingle.ShingleAnalyzerWrapperTest.testOutputUnigramsIfNoShinglesSingleToken
   [junit4]   - org.apache.lucene.analysis.shingle.TestShingleFilterFactory.testOutputUnigramsIfNoShingles
{noformat}

> Track PositionLengthAttribute abuse
> -----------------------------------
>
>                 Key: LUCENE-7708
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7708
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/queryparser, modules/analysis
>            Reporter: Jim Ferenczi
>         Attachments: LUCENE-7708.patch
>
>
> Some token filters uses the position length attribute of the token stream to encode the
number of terms they put in a single token. 
> This breaks the query parsing because it creates disconnected graph. 
> I've tracked down the abusive case to 2 candidates:
> * ShingleFilter which sets the position length attribute to the length of the shingle.
> * CJKBigramFilter which always sets the position length attribute to 2.
> I don't think these filters should set the position length at all so the best would be
to remove the attribute from these token filters but this could break BWC.
> Though this is a serious bug since shingles and cjk bigram now produce invalid queries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message