lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-6582) SynonymFilter should generate a correct (or, at least, better) graph
Date Mon, 22 Jun 2015 22:57:02 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-6582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596804#comment-14596804
] 

Robert Muir commented on LUCENE-6582:
-------------------------------------

Can we consider mike's "crazy graph" as a second step? I don't know, it just seems like the
current patch might be an easier-to-digest improvement. Then again I get totally lost in the
synonymfilter :)

There is a lot still to do after even this first step before users get real improvements,
for instance fixing querybuilder for the boolean case (hopefully easier now), and improving
stuff like sandbox/TermAutomatonQuery (maybe?) for the positional case, and so on.

If we spend the work on a hypercorrect graph for synonymfilter I'm just afraid of things getting
lost, queryparser can't make use of it because posinc/poslen semantics become too confusing,
analysis is hard to reason about because posinc/poslen semantics are too confusing, more performance
tradeoffs (like graph re-sausaging at index time), ...


> SynonymFilter should generate a correct (or, at least, better) graph
> --------------------------------------------------------------------
>
>                 Key: LUCENE-6582
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6582
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Ian Ribas
>         Attachments: LUCENE-6582.patch, LUCENE-6582.patch, after.png, after2.png, after3.png,
before.png
>
>
> Some time ago, I had a problem with synonyms and phrase type queries (actually, it was
elasticsearch and I was using a match query with multiple terms and the "and" operator, as
better explained here: https://github.com/elastic/elasticsearch/issues/10394).
> That issue led to some work on Lucene: LUCENE-6400 (where I helped a little with tests)
and  LUCENE-6401. This issue is also related to LUCENE-3843.
> Starting from the discussion on LUCENE-6400, I'm attempting to implement a solution.
Here is a patch with a first step - the implementation to fix "SynFilter to be able to 'make
positions'" (as was mentioned on the [issue|https://issues.apache.org/jira/browse/LUCENE-6400?focusedCommentId=14498554&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14498554]).
In this way, the synonym filter generates a correct (or, at least, better) graph.
> As the synonym matching is greedy, I only had to worry about fixing the position length
of the rules of the current match, no future or past synonyms would "span" over this match
(please correct me if I'm wrong!). It did require more buffering, twice as much.
> The new behavior I added is not active by default, a new parameter has to be passed in
a new constructor for {{SynonymFilter}}. The changes I made do change the token stream generated
by the synonym filter, and I thought it would be better to let that be a voluntary decision
for now.
> I did some refactoring on the code, but mostly on what I had to change for may implementation,
so that the patch was not too hard to read. I created specific unit tests for the new implementation
({{TestMultiWordSynonymFilter}}) that should show how things will be with the new behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message