lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-6691) REBALANCELEADERS needs to change the leader election queue.
Date Mon, 03 Nov 2014 17:33:33 GMT

     [ https://issues.apache.org/jira/browse/SOLR-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Erick Erickson updated SOLR-6691:
---------------------------------
    Description: 
The original code (SOLR-6517) assumed that changes in the clusterstate after issuing a command
to the overseer to change the leader indicated that the leader was successfully changed. Fortunately,
Noble clued me in that this isn't the case and that the potential leader needs to insert itself
in the leader election queue before trigging the change leader command.

Inserting themselves in the front of the queue should probably happen in BALANCESHARDUNIQUE
when the preferredLeader property is assigned as well.

[~noble.paul] Do evil things happen if a node joins at the head but it's _already_ in the
queue? These ephemeral nodes in the queue are watching each other. So if node1 is the leader
you have
node1 <- node2 <- node3 <- node4
where <- means "watches".

Now, if node3 puts itself at the head of the list, you have
{{code}
node1 <- node2
           <- node3 <- node4
{{code}}

I _think_ when I was looking at this it all "just worked". 
1> node 1 goes down. Nodes 2 and 3 duke it out but there's code to insure that node3 becomes
the leader and node2 inserts itself at then end so it's watching node 4.

2> node 2 goes down, nobody gets notified and it doesn't matter.

3> node 3 goes down, node 4 gets notified and starts watching node 2 by inserting itself
at the end of the list.

4> node 4 goes down, nobody gets notified and it doesn't matter.

  was:
The original code (SOLR-6517) assumed that changes in the clusterstate after issuing a command
to the overseer to change the leader indicated that the leader was successfully changed. Fortunately,
Noble clued me in that this isn't the case and that the potential leader needs to insert itself
in the leader election queue before trigging the change leader command.

Inserting themselves in the front of the queue should probably happen in BALANCESHARDUNIQUE
when the preferredLeader property is assigned as well.

[~noble.paul] Do evil things happen if a node joins at the head but it's _already_ in the
queue? These ephemeral nodes in the queue are watching each other. So if node1 is the leader
you have
node1 <- node2 <- node3 <- node4
where <- means "watches".

Now, if node3 puts itself at the head of the list, you have
node1 <- node2
           <- node3 <- node4

I _think_ when I was looking at this it all "just worked". 
1> node 1 goes down. Nodes 2 and 3 duke it out but there's code to insure that node3 becomes
the leader and node2 inserts itself at then end so it's watching node 4.

2> node 2 goes down, nobody gets notified and it doesn't matter.

3> node 3 goes down, node 4 gets notified and starts watching node 2 by inserting itself
at the end of the list.

4> node 4 goes down, nobody gets notified and it doesn't matter.


> REBALANCELEADERS needs to change the leader election queue.
> -----------------------------------------------------------
>
>                 Key: SOLR-6691
>                 URL: https://issues.apache.org/jira/browse/SOLR-6691
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>
> The original code (SOLR-6517) assumed that changes in the clusterstate after issuing
a command to the overseer to change the leader indicated that the leader was successfully
changed. Fortunately, Noble clued me in that this isn't the case and that the potential leader
needs to insert itself in the leader election queue before trigging the change leader command.
> Inserting themselves in the front of the queue should probably happen in BALANCESHARDUNIQUE
when the preferredLeader property is assigned as well.
> [~noble.paul] Do evil things happen if a node joins at the head but it's _already_ in
the queue? These ephemeral nodes in the queue are watching each other. So if node1 is the
leader you have
> node1 <- node2 <- node3 <- node4
> where <- means "watches".
> Now, if node3 puts itself at the head of the list, you have
> {{code}
> node1 <- node2
>            <- node3 <- node4
> {{code}}
> I _think_ when I was looking at this it all "just worked". 
> 1> node 1 goes down. Nodes 2 and 3 duke it out but there's code to insure that node3
becomes the leader and node2 inserts itself at then end so it's watching node 4.
> 2> node 2 goes down, nobody gets notified and it doesn't matter.
> 3> node 3 goes down, node 4 gets notified and starts watching node 2 by inserting
itself at the end of the list.
> 4> node 4 goes down, nobody gets notified and it doesn't matter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message