lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ishan Chattopadhyaya (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
Date Sat, 29 Oct 2016 07:13:58 GMT

    [ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617617#comment-15617617
] 

Ishan Chattopadhyaya commented on SOLR-5944:
--------------------------------------------



Just discovered another, more common, problem with reordered DBQs and in-place updates working
together. The earlier discussed problem, of resurrecting a document, is very similar. So,
here's a description of both:

SCENARIO 1:
{code}
Imagine, updates on leader are:
ADD	(id=1, updateable_field=1, title="mydoc1", version=100)
INP-UPD	(id=1, updateable_field=2, version=200, prevVersion=100)
DBQ	(q="updateable_field:1", version=300)

The same on the replica (forwarded):
ADD	(id=1, updateable_field=1, title="mydoc1", version=100)
DBQ	(q="updateable_field:1", version=300)
INP-UPD	(id=1, updateable_field=2, version=200, prevVersion=100)

The expected net effect is that no document is deleted, and id=1 document exists with updateable_field=2.
Here, the DBQ was reordered. When they are executed on the replica, the version=200 update
cannot be applied since there is no document with (id=1,prevVersion=100). What is required
is a resurrection of the document that was deleted by the DBQ, so that other stored/indexed
fields are not lost.
{code}

SCENARIO 2:
{code}
Imagine, updates on leader are:
ADD	(id=1, updateable_field=1, title="mydoc1", version=100)
INP-UPD	(id=1, updateable_field=2, version=200, prevVersion=100)
DBQ	(q="id:1", version=300)

The same on the replica (forwarded):
ADD	(id=1, updateable_field=1, title="mydoc1", version=100)
DBQ	(q="id:1", version=300)
INP-UPD	(id=1, updateable_field=2, version=200, prevVersion=100)

The expected net effect is that the document with id=1 be deleted. But again, the DBQ is reordered.
When executed on replica, update version=200 cannot be applied, since the id=1 document has
been deleted. What is required is for this update (version=200) to be dropped silently.
{code}

Scenario 1 is rare, scenario 2 would be more common. At the point when the inplace update
(version=200 in both cases) is applied, the replica has no way to know if the update requires
a resurrection of the document, or requires to be dropped.

Till now, I hadn't considered scenario 2, but for the rare scenario 1, I resorted to throwing
an error so as to throw the replica in LIR. Clearly, in view of scenario 2, this looks like
a bad idea. Here are two potential solutions that come to mind:
Solution 1:
{code}
In a replica, while applying an in-place update, if the required prevVersion update cannot
be found in tlog or index (due to these reordered DBQs), then fetch from the leader an update
that contains the full document with the version (for which the update failed at replica).
Downside to this approach is that unstored/non-dv fields will get dropped (as is the case
with regular atomic updates today).
{code}
Solution 2:
{code}
Ensure that DBQs are never reordered from leader -> replica. One approach can be SOLR-8148.
Another could be to block, on the leader, all updates newer than a DBQ, that has been sent
through a different thread, until the DBQ is processed on leader and all the replicas, and
only then process the other updates.
{code}
Solution 1 seems easier to implement now than solution 2, but solution 2 (if implemented correctly)
seems cleaner. Any thoughts?


> Support updates of numeric DocValues
> ------------------------------------
>
>                 Key: SOLR-5944
>                 URL: https://issues.apache.org/jira/browse/SOLR-5944
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ishan Chattopadhyaya
>            Assignee: Shalin Shekhar Mangar
>         Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, hoss.62D328FA1DEA57FD.fail.txt,
hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice
to have Solr support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message