lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Goodman <richa...@brandwatch.com>
Subject auotscaling tlog+pull and replicationFactor
Date Thu, 02 Apr 2020 14:21:11 GMT
Hi there,

I'm currently using solr v7.7.2 *(I applied the patch SOLR-13674
<https://issues.apache.org/jira/browse/SOLR-13674> to my build to preserve
replica types for triggers)* and trying to set up a cluster with the
combination of TLOG+PULL replicas and utilising the solr autoscaling
feature to maintain the stability of the cluster.

I'm getting confused though with the documentation.

When creating a collection, if you specify "replicationFactor=3" then it'll
create 3 NRT replicas, and from my understanding, this information is
preserved within autoscaling, i.e. if a node went down, it would attempt to
add another replica to preserve the replicationFactor.

However, because I'm using TLOG + PULL, I don't add the
"replicationFactor=3" into my collection creation, otherwise it ends up
creating a collection with 3 NRT replicas, 1 TLOG and 2 PULL replicas,
which of course I do not want.

Instead, I add "&tlogReplicas=1&pullReplicas=2" which satisfies. However,
when it comes to me creating policies, I'm not getting violations when
expected.

I created the following policies:
{
  "replica": "<2",
  "shard": "#EACH",
  "type": "TLOG",
  "node": "#ANY"
},
{
  "replica": "<3",
  "shard": "#EACH",
  "type": "PULL",
  "node": "#ANY"
}

And there appears to be 0 violations, however, when I add another replica,
there are still 0 violations despite there now being 4 replicas for a given
shard of a collection;

    "violations":[],
    "config":{
      "cluster-preferences":[{
          "minimize":"cores",
          "precision":1}
        ,{
          "maximize":"freedisk"}],
      "cluster-policy":[{
          "replica":"<2",
          "shard":"#EACH",
          "type":"TLOG",
          "node":"#ANY"}
        ,{
          "replica":"<3",
          "shard":"#EACH",
          "type":"PULL",
          "node":"#ANY"}]}},

And here is the information about a collection-shard in which I added an
extra replica to which I expected there to be a violation:
{
  "replica_p35": {
    "core": "collection_four_shard1_replica_p35",
    "shard": "shard1",
    "collection": "collection_four",
    "state": "active",
    "shard_name": "collection_four_shard1",
    "num_shards": 6,
    "type": "PULL"
  },
  "replica_p31": {
    "core": "collection_four_shard1_replica_p31",
    "shard": "shard1",
    "collection": "collection_four",
    "state": "active",
    "shard_name": "collection_four_shard1",
    "num_shards": 6,
    "type": "PULL"
  },
  "replica_p87": {
    "core": "collection_four_shard1_replica_p87",
    "shard": "shard1",
    "collection": "collection_four",
    "state": "active",
    "shard_name": "collection_four_shard1",
    "num_shards": 6,
    "type": "PULL"
  },
  "replica_t75": {
    "core": "collection_four_shard1_replica_t75",
    "shard": "shard1",
    "collection": "collection_four",
    "state": "active",
    "shard_name": "collection_four_shard1",
    "type": "TLOG",
    "leader": "true"
  }
}

Am I missing something to preserve replicationFactor for a collection, and
trying to make collections have 1 TLOG replica and 2 PULL replicas?

I tried adding the following
{"replicas": "<4", "shard":"#EACH", "node": "#ANY"}

However, still no luck

Equally, how would I then go about setting up triggers to only create a
PULL if a PULL goes down, and equally if the TLOG goes down? Would having a
trigger for each type be needed?

Any guidance on this would be greatly appreciated
Thanks,
-- 

Richard Goodman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message