lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-5468) Option to enforce a majority quorum approach to accepting updates in SolrCloud
Date Mon, 05 May 2014 17:20:27 GMT

    [ https://issues.apache.org/jira/browse/SOLR-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989724#comment-13989724
] 

Mark Miller edited comment on SOLR-5468 at 5/5/14 5:20 PM:
-----------------------------------------------------------

I won't be able to catch up to this in a competent way for a week or two but a quick comment:

First, this is def a hard problem. It's something I have thought a bit about, but have not
cracked yet.

bq.  However, that's exactly what happens automatically 

I'm not sure 'exactly' fits. I think it's probably important to let the client know that an
update only made it to one replica for example. That mean's you are in a position to possibly
lose the update. One possibility is to define the fail the same way as a leader fail in the
middle of your update. You don't know what happened in that case - the doc may be in or not.
If we do the same here, the client will know, hey, this didn't make it to 2 or 3 replicas
- it may be in the cluster, but we can't count on it's durability to the level we requested.
The client can then choose how to handle this - accept what happened or take another action.

Just spit balling at a conference, but I think there is a way to define the semantics here
so that it's easier on us, but still gives the client the info they need to understand how
durable that update was. This would not be the only case a fail does not mean the update is
for sure not in the cluster - you can't get around that on fails of the leader in the middle
of an update anyway.


was (Author: markrmiller@gmail.com):
I won't be able to catch up to this in a competent for a week or two but a quick comment:

First, this is def a hard problem. It's something I have thought a bit about, but have not
cracked yet.

bq.  However, that's exactly what happens automatically 

I'm not sure 'exactly' fits. I think it's probably important to let the client know that an
update only made it to one replica for example. That mean's you are in a position to possibly
lose the update. One possibility is to define the fail the same way as a leader fail in the
middle of your update. You don't know what happened in that case - the doc may be in or not.
If we do the same here, the client will know, hey, this didn't make it to 2 or 3 replicas
- it may be in the cluster, but we can't count on it's durability to the level we requested.
The client can then choose how to handle this - accept what happened or take another action.

Just spit balling at a conference, but I think there is a way to define the semantics here
so that it's easier on us, but still gives the client the info they need to understand how
durable that update was. This would not be the only case a fail does not mean the update is
for sure not in the cluster - you can't get around that on fails of the leader in the middle
of an update anyway.

> Option to enforce a majority quorum approach to accepting updates in SolrCloud
> ------------------------------------------------------------------------------
>
>                 Key: SOLR-5468
>                 URL: https://issues.apache.org/jira/browse/SOLR-5468
>             Project: Solr
>          Issue Type: New Feature
>          Components: SolrCloud
>    Affects Versions: 4.5
>         Environment: All
>            Reporter: Timothy Potter
>            Assignee: Timothy Potter
>            Priority: Minor
>         Attachments: SOLR-5468.patch
>
>
> I've been thinking about how SolrCloud deals with write-availability using in-sync replica
sets, in which writes will continue to be accepted so long as there is at least one healthy
node per shard.
> For a little background (and to verify my understanding of the process is correct), SolrCloud
only considers active/healthy replicas when acknowledging a write. Specifically, when a shard
leader accepts an update request, it forwards the request to all active/healthy replicas and
only considers the write successful if all active/healthy replicas ack the write. Any down
/ gone replicas are not considered and will sync up with the leader when they come back online
using peer sync or snapshot replication. For instance, if a shard has 3 nodes, A, B, C with
A being the current leader, then writes to the shard will continue to succeed even if B &
C are down.
> The issue is that if a shard leader continues to accept updates even if it loses all
of its replicas, then we have acknowledged updates on only 1 node. If that node, call it A,
then fails and one of the previous replicas, call it B, comes back online before A does, then
any writes that A accepted while the other replicas were offline are at risk to being lost.

> SolrCloud does provide a safe-guard mechanism for this problem with the leaderVoteWait
setting, which puts any replicas that come back online before node A into a temporary wait
state. If A comes back online within the wait period, then all is well as it will become the
leader again and no writes will be lost. As a side note, sys admins definitely need to be
made more aware of this situation as when I first encountered it in my cluster, I had no idea
what it meant.
> My question is whether we want to consider an approach where SolrCloud will not accept
writes unless there is a majority of replicas available to accept the write? For my example,
under this approach, we wouldn't accept writes if both B&C failed, but would if only C
did, leaving A & B online. Admittedly, this lowers the write-availability of the system,
so may be something that should be tunable?
> From Mark M: Yeah, this is kind of like one of many little features that we have just
not gotten to yet. I’ve always planned for a param that let’s you say how many replicas
an update must be verified on before responding success. Seems to make sense to fail that
type of request early if you notice there are not enough replicas up to satisfy the param
to begin with.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message