lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Cline <aaron.cl...@gmail.com>
Subject AutoScaling Solr on AWS
Date Thu, 03 Jan 2019 17:46:36 GMT
Solr Version 7.3.1
Java Version 1.8.0_151

I'm trying to get solrcloud to autoscale when a new node is added to the
cluster and balance the existing replicas across the new node accordingly.
I'm running into some kind of odd error during the compute_plan action.
I'm hoping someone here will point me in the right direction.  Please let
me know if I need to provide more information.

Here is the log of the error from the solr leader:

2019-01-03 17:23:10.268 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node19
2019-01-03 17:23:10.276 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node9
2019-01-03 17:23:10.283 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node13
2019-01-03 17:23:10.292 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node3
2019-01-03 17:23:10.301 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-gsr-content0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node7
2019-01-03 17:23:10.309 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node17
2019-01-03 17:23:10.318 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node9
2019-01-03 17:23:10.329 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-gsr-content0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node11
2019-01-03 17:23:10.337 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-gsr-content0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node3
2019-01-03 17:23:10.345 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node13
2019-01-03 17:23:10.353 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node17
2019-01-03 17:23:10.360 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node13
2019-01-03 17:23:10.367 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node20
2019-01-03 17:23:10.375 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node20
2019-01-03 17:23:10.382 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders1&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node5
2019-01-03 17:23:10.389 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node5
2019-01-03 17:23:10.396 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node9
2019-01-03 17:23:10.403 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-gsr-content0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node17
2019-01-03 17:23:10.411 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-gsr-content0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node20
2019-01-03 17:23:10.418 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node13
2019-01-03 17:23:10.424 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node17
2019-01-03 17:23:10.431 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node9
2019-01-03 17:23:10.438 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node13
2019-01-03 17:23:10.445 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node20
2019-01-03 17:23:10.452 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-customers0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node5
2019-01-03 17:23:10.458 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node17
2019-01-03 17:23:10.466 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ComputePlanAction Computed Plan:
action=MOVEREPLICA&collection=blc-fulfillment-orders0&targetNode=ip-10-228-7-27.local:8983_solr&inPlaceMove=true&replica=core_node20
2019-01-03 17:23:10.466 INFO
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 1861149605
sessionWrapper.createTime 1861149397601432, this.sessionWrapper.createTime
1861149397601432
2019-01-03 17:23:10.466 WARN
(AutoscalingActionExecutor-7-thread-1-processing-n:ip-10-228-12-123.local:8983_solr)
[   ] o.a.s.c.a.ScheduledTriggers Exception executing actions
java.lang.Exception: Error executing action: compute_plan for trigger
event: {
  "id":"69ca688d47f24Tbojh9oy9a8xwifvk2wfynxs5k",
  "source":"node_added_trigger",
  "eventTime":1861088934395684,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[1861088934395684],
    "_enqueue_time_":1861148945631973,
    "nodeNames":["ip-10-228-7-27.local:8983_solr"]}}
        at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Unexpected exception while
processing event: {
  "id":"69ca688d47f24Tbojh9oy9a8xwifvk2wfynxs5k",
  "source":"node_added_trigger",
  "eventTime":1861088934395684,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[1861088934395684],
    "_enqueue_time_":1861148945631973,
    "nodeNames":["ip-10-228-7-27.local:8983_solr"]}}
        at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:144)
        at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:304)
        ... 6 more
Caused by: java.lang.IllegalArgumentException: Comparison method violates
its general contract!
        at java.util.TimSort.mergeLo(TimSort.java:777)
        at java.util.TimSort.mergeAt(TimSort.java:514)
        at java.util.TimSort.mergeCollapse(TimSort.java:441)
        at java.util.TimSort.sort(TimSort.java:245)
        at java.util.Arrays.sort(Arrays.java:1512)
        at java.util.ArrayList.sort(ArrayList.java:1460)
        at
org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:46)
        at
org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at
org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        ... 7 more



Here's the config from the /api/cluster/autoscaling endpoint:

{
  "responseHeader": {
    "status": 0,
    "QTime": 0
  },
  "cluster-preferences": [
    {
      "minimize": "cores",
      "precision": 1
    },
    {
      "maximize": "freedisk"
    }
  ],
  "triggers": {
    ".auto_add_replicas": {
      "name": ".auto_add_replicas",
      "event": "nodeLost",
      "waitFor": 120,
      "actions": [
        {
          "name": "auto_add_replicas_plan",
          "class": "solr.AutoAddReplicasPlanAction"
        },
        {
          "name": "execute_plan",
          "class": "solr.ExecutePlanAction"
        }
      ],
      "enabled": true
    },
    "node_added_trigger": {
      "event": "nodeAdded",
      "waitFor": 60,
      "actions": [
        {
          "name": "compute_plan",
          "class": "solr.ComputePlanAction"
        },
        {
          "name": "execute_plan",
          "class": "solr.ExecutePlanAction"
        }
      ],
      "enabled": true
    }
  },
  "listeners": {
    ".auto_add_replicas.system": {
      "trigger": ".auto_add_replicas",
      "afterAction": [],
      "stage": [
        "STARTED",
        "ABORTED",
        "SUCCEEDED",
        "FAILED",
        "BEFORE_ACTION",
        "AFTER_ACTION",
        "IGNORED"
      ],
      "class": "org.apache.solr.cloud.autoscaling.SystemLogListener",
      "beforeAction": []
    },
    "node_added_trigger.system": {
      "trigger": "node_added_trigger",
      "afterAction": [],
      "stage": [
        "STARTED",
        "ABORTED",
        "SUCCEEDED",
        "FAILED",
        "BEFORE_ACTION",
        "AFTER_ACTION",
        "IGNORED"
      ],
      "class": "org.apache.solr.cloud.autoscaling.SystemLogListener",
      "beforeAction": []
    }
  },
  "properties": {},
  "WARNING": "This response format is experimental.  It is likely to change
in the future."
}

All I've added beyond the auto_add_replica settings is the:
    "node_added_trigger": {
      "event": "nodeAdded",
      "waitFor": 60,
      "actions": [
        {
          "name": "compute_plan",
          "class": "solr.ComputePlanAction"
        },
        {
          "name": "execute_plan",
          "class": "solr.ExecutePlanAction"
        }
      ],
      "enabled": true
    }

section, so I'm guessing I'm missing something else perhaps, but I'm not
sure what that might be.

Thanks for any help.

Aaron

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message