lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-6491) Umbrella JIRA for managing the leader assignments
Date Tue, 16 Sep 2014 19:25:34 GMT

    [ https://issues.apache.org/jira/browse/SOLR-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136004#comment-14136004
] 

Erick Erickson edited comment on SOLR-6491 at 9/16/14 7:25 PM:
---------------------------------------------------------------

bq: A node may be preferred for sometime and after sometime it can change
True. That's where the SOLR-6513 comes in. Remember also that this is a _preference_, not
an absolute map.

bq: After a new leader election happens overseer does a quick check on all the nodes where
the shard has replicas
One situation we're talking about here is one in which the pathology has already happened.
For illustration, consider the following:

I have 10 shards and 10 nodes. I have 100 replicas, i.e. each node has one replica for every
shard.
All the nodes are shut down.
I start 1 node. All 10 leaders are elected on that node.
Now I bring up my other 9 machines.

No leader election goes on here as the other 9 nodes come up. So "After a new leader election
happens overseer does a quick check on all the nodes where the shard has replicas, and find
out if there is a more suitable node for that shard in the cluster" doesn't apply since there's
no leader election going on. I suppose we could just fire a "balance leaders" command, but
that kind of pre-supposes a symmetric cluster.

I'm reluctant to try to do everything automatically. Consider an asymmetric cluster, both
in terms of hardware and collection characteristics. You have machines of varying power hosting
(perhaps many) collections with different usage patterns. I don't see how to replace the operations
folks' knowledge of all this with an auto-assignment process; the "and find out if there is
a more suitable node" above. Suitable by what criteria? To do this really well, we'd need
to have a language to define capabilities for both collections and nodes, and a heuristic
to factor all that in auto-leader election.

If we do take a stab at figuring this all out automatically, it seems that the preferredLeader
mechanism could be the way this is realized; auto-assign the roles, then trigger the re-election
process. With room for the ops people to tweak it as they see fit.

Now all _that_ said, maybe I'm over-thinking the problem. It might be "good enough" to just
not bother with the preferredLeader role and have a "rebalance leaders now" command. And trust
the operations people to distribute their collections appropriately. By "appropriately", I'm
thinking of assigning the collections across the cluster such that the leaders will be acceptably
distributed if we just balance them within a collection.

BTW, I think the idea of more intelligently electing leaders is a good one, and this in no
way means we shouldn't do that too. I'm concerned more intelligence there is only part of
the total solution though.


was (Author: erickerickson):
bq: A node may be preferred for sometime and after sometime it can change
True. That's where the SOLR-6513 comes in. Remember also that this is a _preference_, not
an absolute map.

bq: After a new leader election happens overseer does a quick check on all the nodes where
the shard has replicas
One situation we're talking about here is one in which the pathology has already happened.
For illustration, consider the following:

I have 10 shards and 10 nodes. I have 100 replicas, i.e. each node has one replica for every
shard.
All the nodes are shut down.
I start 1 node. All 10 leaders are elected on that node.
Now I bring up my other 9 machines.

No leader election goes on here as the other 9 nodes come up. So "After a new leader election
happens overseer does a quick check on all the nodes where the shard has replicas, and find
out if there is a more suitable node for that shard in the cluster" doesn't apply since there's
no leader election going on. I suppose we could just fire a "balance leaders" command, but
that kind of pre-supposes a symmetric cluster.

I'm reluctant to try to do everything automatically. Consider an asymmetric cluster, both
in terms of hardware and collection characteristics. You have machines of varying power hosting
(perhaps many) collections with different usage patterns. I don't see how to replace the operations
folks' knowledge of all this with an auto-assignment process; the "and find out if there is
a more suitable node" above. Suitable by what criteria? To do this really well, we'd need
to have a language to define capabilities for both collections and nodes, and a heuristic
to factor all that in auto-leader election.

If we do take a stab at figuring this all out automatically, it seems that the preferredLeader
mechanism could be the way this is realized; auto-assign the roles, then trigger the re-election
process. With room for the ops people to tweak it as they see fit.

Now all _that_ said, maybe I'm over-thinking the problem. It might be "good enough" to just
not bother with the preferredLeader role and have a "rebalance leaders now" command. And trust
the operations people to distribute their collections appropriately. By "appropriately", I'm
thinking of assigning the collections across the cluster such that the leaders will be acceptably
distributed if we just balance them within a collection.

> Umbrella JIRA for managing the leader assignments
> -------------------------------------------------
>
>                 Key: SOLR-6491
>                 URL: https://issues.apache.org/jira/browse/SOLR-6491
>             Project: Solr
>          Issue Type: Improvement
>    Affects Versions: 4.11, 5.0
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>
> Leaders can currently get out of balance due to the sequence of how nodes are brought
up in a cluster. For very good reasons shard leadership cannot be permanently assigned.
> However, it seems reasonable that a sys admin could optionally specify that a particular
node be the _preferred_ leader for a particular collection/shard. During leader election,
preference would be given to any node so marked when electing any leader.
> So the proposal here is to add another role for preferredLeader to the collections API,
something like
> ADDROLE?role=preferredLeader&collection=collection_name&shard=shardId
> Second, it would be good to have a new collections API call like ELECTPREFERREDLEADERS?collection=collection_name
> (I really hate that name so far, but you see the idea). That command would (asynchronously?)
make an attempt to transfer leadership for each shard in a collection to the leader labeled
as the preferred leader by the new ADDROLE role.
> I'm going to start working on this, any suggestions welcome!
> This will subsume several other JIRAs, I'll link them momentarily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message