lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS-EA] Lucene-Solr-8.x-Linux (64bit/jdk-12-ea+23) - Build # 36 - Unstable!
Date Wed, 16 Jan 2019 00:08:45 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/36/
Java: 64bit/jdk-12-ea+23 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([FCBBB2AC9FCECAC8:350EF00296A90C3D]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:718)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 13353 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_FCBBB2AC9FCECAC8-001/init-core-data-001
   [junit4]   2> 475799 WARN  (SUITE-TestSimTriggerIntegration-seed#[FCBBB2AC9FCECAC8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 475800 INFO  (SUITE-TestSimTriggerIntegration-seed#[FCBBB2AC9FCECAC8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 475800 INFO  (SUITE-TestSimTriggerIntegration-seed#[FCBBB2AC9FCECAC8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 475801 INFO  (SUITE-TestSimTriggerIntegration-seed#[FCBBB2AC9FCECAC8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 475803 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 475811 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10009_solr
   [junit4]   2> 475812 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 475812 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 475814 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 475814 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 475815 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 475815 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 475815 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 475817 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10009_solr]
   [junit4]   2> 475817 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 475818 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 475818 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 475822 DEBUG (simCloudManagerPool-4643-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 475822 DEBUG (simCloudManagerPool-4643-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 475855 DEBUG (ScheduledTrigger-4644-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 475856 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 475856 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 475856 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 475864 DEBUG (simCloudManagerPool-4643-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 475864 DEBUG (simCloudManagerPool-4643-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 475865 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 475866 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 475866 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10009_solr]
   [junit4]   2> 475866 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 475866 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 475866 DEBUG (ScheduledTrigger-4644-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 475877 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 475888 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 475890 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider  -- created marker: /autoscaling/nodeLost/127.0.0.1:10011_solr
   [junit4]   2> 475897 DEBUG (ScheduledTrigger-4644-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 3
   [junit4]   2> 475897 DEBUG (ScheduledTrigger-4644-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
   [junit4]   2> 475908 DEBUG (ScheduledTrigger-4644-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2 and last live nodes: 3
   [junit4]   2> 475908 DEBUG (ScheduledTrigger-4644-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
   [junit4]   2> 475913 DEBUG (simCloudManagerPool-4643-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 475914 DEBUG (simCloudManagerPool-4643-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 475914 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 475914 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 475914 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10009_solr]
   [junit4]   2> 475914 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 475914 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 475914 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 475917 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 475935 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 475935 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10011_solr]
   [junit4]   2> 475935 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"903bce5fef2deTb7796dwzlmgs1cgletigyjndm",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":2537384637821662,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537384637821662],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 475935 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_restore_trigger]
   [junit4]   2> 475935 DEBUG (ScheduledTrigger-4644-thread-2) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903bce5fef2deTb7796dwzlmgs1cgletigyjndm",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":2537384637821662,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537384637821662],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537385996198812,
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 475941 DEBUG (AutoscalingActionExecutor-4645-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"903bce5fef2deTb7796dwzlmgs1cgletigyjndm",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":2537384637821662,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537384637821662],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537385996198812,
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 475942 DEBUG (AutoscalingActionExecutor-4645-thread-1) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903bce5fef2deTb7796dwzlmgs1cgletigyjndm",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":2537384637821662,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537384637821662],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537385996198812,
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 475942 DEBUG (AutoscalingActionExecutor-4645-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_restore_trigger after 100ms
   [junit4]   2> 475942 DEBUG (AutoscalingActionExecutor-4645-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 475942 DEBUG (AutoscalingActionExecutor-4645-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 29 ms for event id=903bce5fef2deTb7796dwzlmgs1cgletigyjndm
   [junit4]   2> 475942 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 475942 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 475943 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 475943 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 475944 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 475944 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 475944 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 475945 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 475947 INFO  (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 475963 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10012_solr
   [junit4]   2> 475964 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 475964 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 475964 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 475964 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 475981 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 475981 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 475981 INFO  (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 475999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10012_solr]
   [junit4]   2> 476022 DEBUG (simCloudManagerPool-4649-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476022 DEBUG (simCloudManagerPool-4649-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 476027 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476033 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 1
   [junit4]   2> 476034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 476035 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476035 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 476047 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476048 DEBUG (simCloudManagerPool-4649-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection collection1, currentVersion=1
   [junit4]   2> 476048 DEBUG (simCloudManagerPool-4649-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 476048 DEBUG (simCloudManagerPool-4649-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 476060 DEBUG (simCloudManagerPool-4649-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 476062 DEBUG (simCloudManagerPool-4649-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 476071 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476092 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476093 DEBUG (simCloudManagerPool-4649-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 476094 DEBUG (simCloudManagerPool-4649-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 476112 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476132 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476152 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476173 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476174 DEBUG (simCloudManagerPool-4649-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for collection1 / shard1 (currentVersion=4): {"core_node1":{
   [junit4]   2>     "core":"collection1_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"collection1",
   [junit4]   2>     "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 476174 DEBUG (simCloudManagerPool-4649-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection collection1, currentVersion=4
   [junit4]   2> 476185 DEBUG (simCloudManagerPool-4649-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 476185 DEBUG (simCloudManagerPool-4649-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 476185 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=5
   [junit4]   2> 476186 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 476193 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476200 DEBUG (simCloudManagerPool-4649-thread-7) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476201 DEBUG (simCloudManagerPool-4649-thread-7) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 476201 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 476201 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476201 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 476210 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10013_solr
   [junit4]   2> 476210 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10012_solr
   [junit4]   2> 476215 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476219 DEBUG (simCloudManagerPool-4649-thread-8) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476219 DEBUG (simCloudManagerPool-4649-thread-8) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 476223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 476223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 476239 DEBUG (ScheduledTrigger-4650-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476239 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10013_solr
   [junit4]   2> 476239 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n2:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476240 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10012_solr
   [junit4]   2> 476240 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n1:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476241 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ## Collection: collection1
   [junit4]   2> 476241 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##  - shard1
   [junit4]   2> 476241 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n2  250.0
   [junit4]   2> 476241 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n1  250.0
   [junit4]   2> 476252 DEBUG (ScheduledTrigger-4650-thread-4) [    ] o.a.s.c.a.SearchRateTrigger -- collection1.shard1: totalShardRate=500.0, shardRate=250.0
   [junit4]   2> 476263 DEBUG (ScheduledTrigger-4650-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476273 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10013_solr
   [junit4]   2> 476273 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n2:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10012_solr
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n1:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ## Collection: collection1
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ##  - shard1
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n2  250.0
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n1  250.0
   [junit4]   2> 476274 DEBUG (ScheduledTrigger-4650-thread-1) [    ] o.a.s.c.a.SearchRateTrigger -- collection1.shard1: totalShardRate=500.0, shardRate=250.0
   [junit4]   2> 476283 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10013_solr
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n2:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10012_solr
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n1:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ## Collection: collection1
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ##  - shard1
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n2  250.0
   [junit4]   2> 476294 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n1  250.0
   [junit4]   2> 476295 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger -- collection1.shard1: totalShardRate=500.0, shardRate=250.0
   [junit4]   2> 476304 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476315 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10013_solr
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n2:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10012_solr
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.collection1.shard1.replica_n1:QUERY./select.requestTimes:1minRate	250
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ## Collection: collection1
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ##  - shard1
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n2  250.0
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger ##     collection1_shard1_replica_n1  250.0
   [junit4]   2> 476316 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SearchRateTrigger -- collection1.shard1: totalShardRate=500.0, shardRate=250.0
   [junit4]   2> 476321 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"903c0c64e1402Tb7796dwzlmgs1cgletigyjndo",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":2537401286005762,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10013_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"collection1_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":250.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10013_solr":250.0,
   [junit4]   2>       "127.0.0.1:10012_solr":250.0},
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}}]}}
   [junit4]   2> 476321 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, search_rate_trigger]
   [junit4]   2> 476325 DEBUG (ScheduledTrigger-4650-thread-2) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903c0c64e1402Tb7796dwzlmgs1cgletigyjndo",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":2537401286005762,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10013_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"collection1_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":250.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10013_solr":250.0,
   [junit4]   2>       "127.0.0.1:10012_solr":250.0},
   [junit4]   2>     "_enqueue_time_":2537405295062362,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}}]}}
   [junit4]   2> 476329 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"903c0c64e1402Tb7796dwzlmgs1cgletigyjndo",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":2537401286005762,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10013_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"collection1_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":250.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10013_solr":250.0,
   [junit4]   2>       "127.0.0.1:10012_solr":250.0},
   [junit4]   2>     "_enqueue_time_":2537405295062362,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}}]}}
   [junit4]   2> 476330 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"903c0c64e1402Tb7796dwzlmgs1cgletigyjndo",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":2537401286005762,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10013_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"collection1_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":250.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10013_solr":250.0,
   [junit4]   2>       "127.0.0.1:10012_solr":250.0},
   [junit4]   2>     "_enqueue_time_":2537405295062362,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}}]}} with context properties: {AFTER_ACTION=[start], BEFORE_ACTION=[start, compute]}
   [junit4]   2> 476330 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10012_solr&type=NRT
   [junit4]   2> 476330 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10013_solr&type=NRT
   [junit4]   2> 476330 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10013_solr&type=NRT
   [junit4]   2> 476331 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ExecutePlanAction -- processing event: {
   [junit4]   2>   "id":"903c0c64e1402Tb7796dwzlmgs1cgletigyjndo",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":2537401286005762,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10013_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"collection1_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "__start__":3,
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":250.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10013_solr":250.0,
   [junit4]   2>       "127.0.0.1:10012_solr":250.0},
   [junit4]   2>     "_enqueue_time_":2537405295062362,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}}]}} with context properties: {operations=[{
   [junit4]   2>   "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica",
   [junit4]   2>   "method":"GET",
   [junit4]   2>   "params.action":"ADDREPLICA",
   [junit4]   2>   "params.collection":"collection1",
   [junit4]   2>   "params.shard":"shard1",
   [junit4]   2>   "params.node":"127.0.0.1:10012_solr",
   [junit4]   2>   "params.type":"NRT"}, {
   [junit4]   2>   "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica",
   [junit4]   2>   "method":"GET",
   [junit4]   2>   "params.action":"ADDREPLICA",
   [junit4]   2>   "params.collection":"collection1",
   [junit4]   2>   "params.shard":"shard1",
   [junit4]   2>   "params.node":"127.0.0.1:10013_solr",
   [junit4]   2>   "params.type":"NRT"}, {
   [junit4]   2>   "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica",
   [junit4]   2>   "method":"GET",
   [junit4]   2>   "params.action":"ADDREPLICA",
   [junit4]   2>   "params.collection":"collection1",
   [junit4]   2>   "params.shard":"shard1",
   [junit4]   2>   "params.node":"127.0.0.1:10013_solr",
   [junit4]   2>   "params.type":"NRT"}], AFTER_ACTION=[start, compute], BEFORE_ACTION=[start, compute, execute]}
   [junit4]   2> 476331 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10012_solr&type=NRT
   [junit4]   2> 476338 INFO  (simCloudManagerPool-4649-thread-9) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10012_solr for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 476338 INFO  (simCloudManagerPool-4649-thread-9) [    ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 476349 DEBUG (simCloudManagerPool-4649-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=6
   [junit4]   2> 476349 DEBUG (simCloudManagerPool-4649-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 476387 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10013_solr&type=NRT
   [junit4]   2> 476390 INFO  (simCloudManagerPool-4649-thread-12) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10013_solr for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 476390 INFO  (simCloudManagerPool-4649-thread-12) [    ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 476401 DEBUG (simCloudManagerPool-4649-thread-12) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=7
   [junit4]   2> 476401 DEBUG (simCloudManagerPool-4649-thread-12) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 476429 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10013_solr&type=NRT
   [junit4]   2> 476439 INFO  (simCloudManagerPool-4649-thread-15) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10013_solr for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 476439 INFO  (simCloudManagerPool-4649-thread-15) [    ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 476449 DEBUG (simCloudManagerPool-4649-thread-15) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=8
   [junit4]   2> 476450 DEBUG (simCloudManagerPool-4649-thread-15) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 476493 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903c0c64e1402Tb7796dwzlmgs1cgletigyjndo",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":2537401286005762,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10013_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"collection1_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"collection1",
   [junit4]   2>           "node_name":"127.0.0.1:10012_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":250.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "__start__":3,
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":250.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10013_solr":250.0,
   [junit4]   2>       "127.0.0.1:10012_solr":250.0},
   [junit4]   2>     "_enqueue_time_":2537405295062362,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}},
   [junit4]   2>       {
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"collection1",
   [junit4]   2>               "second":"shard1"}]}}]}}
   [junit4]   2> 476493 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 476494 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: search_rate_trigger after 100ms
   [junit4]   2> 476494 DEBUG (AutoscalingActionExecutor-4651-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 8283 ms for event id=903c0c64e1402Tb7796dwzlmgs1cgletigyjndo
   [junit4]   2> 476502 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 476502 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 476502 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 476502 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 476503 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 476511 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 476511 DEBUG (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 476512 INFO  (TEST-TestSimTriggerIntegration.testSearchRate-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
   [junit4]   2> 476523 INFO  (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
   [junit4]   2> 476531 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10014_solr
   [junit4]   2> 476535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 476536 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 476536 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 476536 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 476562 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 476562 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 476562 INFO  (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 476575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10014_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 476579 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476579 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 476583 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476603 DEBUG (simCloudManagerPool-4655-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476604 DEBUG (simCloudManagerPool-4655-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 476614 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 476647 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476647 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 476648 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476656 DEBUG (simCloudManagerPool-4655-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476656 DEBUG (simCloudManagerPool-4655-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 476656 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 476656 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 476656 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 - Initial livenodes: [127.0.0.1:10014_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 476656 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 476656 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476656 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 476657 DEBUG (ScheduledTrigger-4656-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 476657 DEBUG (ScheduledTrigger-4656-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 476668 DEBUG (ScheduledTrigger-4656-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476670 DEBUG (simCloudManagerPool-4655-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476671 DEBUG (simCloudManagerPool-4655-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 476671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 476671 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 476671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 - Initial livenodes: [127.0.0.1:10014_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 476671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 476671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 476671 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 476671 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 476672 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- creating marker: /autoscaling/nodeAdded/127.0.0.1:10016_solr
   [junit4]   2> 476677 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 476677 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 476677 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10016_solr at time 2537423098589812
   [junit4]   2> 476677 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10016_solr] added at times [2537423098589812], now=2537423100379612
   [junit4]   2> 476677 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"903c5da6fb274Tb7796dwzlmgs1cgletigyjndq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2537423098589812,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537423098589812],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476678 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger2, node_added_trigger1]
   [junit4]   2> 476678 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903c5da6fb274Tb7796dwzlmgs1cgletigyjndq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2537423098589812,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537423098589812],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537423114819562,
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476699 DEBUG (ScheduledTrigger-4656-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10016_solr
   [junit4]   2> 476707 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"903c5da6fb274Tb7796dwzlmgs1cgletigyjndq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2537423098589812,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537423098589812],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537423114819562,
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476707 INFO  (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration action executed from node_added_trigger1
   [junit4]   2> 476707 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903c5da6fb274Tb7796dwzlmgs1cgletigyjndq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2537423098589812,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537423098589812],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537423114819562,
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476707 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
   [junit4]   2> 476707 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 476707 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 476708 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 24 ms for event id=903c5da6fb274Tb7796dwzlmgs1cgletigyjndq
   [junit4]   2> 476808 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 476808 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10016_solr at time 2537429673259412
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10016_solr] added at times [2537429673259412], now=2537429674662762
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"903c762513194Tb7796dwzlmgs1cgletigyjndr",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":2537429673259412,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537429673259412],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger2, node_added_trigger1]
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903c762513194Tb7796dwzlmgs1cgletigyjndr",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":2537429673259412,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537429673259412],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537429685546662,
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476809 DEBUG (ScheduledTrigger-4656-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10016_solr
   [junit4]   2> 476809 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"903c762513194Tb7796dwzlmgs1cgletigyjndr",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":2537429673259412,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537429673259412],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537429685546662,
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476809 INFO  (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration last action at 2537424597817412 time = 2537429701837962
   [junit4]   2> 476809 INFO  (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration action executed from node_added_trigger2
   [junit4]   2> 476809 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903c762513194Tb7796dwzlmgs1cgletigyjndr",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":2537429673259412,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537429673259412],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537429685546662,
   [junit4]   2>     "nodeNames":["127.0.0.1:10016_solr"]}}
   [junit4]   2> 476810 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 476810 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 476810 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
   [junit4]   2> 476810 DEBUG (AutoscalingActionExecutor-4657-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 22 ms for event id=903c762513194Tb7796dwzlmgs1cgletigyjndr
   [junit4]   2> 476832 DEBUG (simCloudManagerPool-4655-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476832 DEBUG (simCloudManagerPool-4655-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 476832 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 476832 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 476832 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger1 - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10014_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 476832 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476832 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 476833 DEBUG (ScheduledTrigger-4656-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 476850 DEBUG (simCloudManagerPool-4655-thread-6) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476850 DEBUG (simCloudManagerPool-4655-thread-6) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 476850 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 476850 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 476851 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger2 - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10014_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 476851 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476851 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 476851 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 476853 DEBUG (ScheduledTrigger-4656-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 476856 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider  -- created marker: /autoscaling/nodeLost/127.0.0.1:10016_solr
   [junit4]   2> 476857 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 476857 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 476857 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 476857 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 476857 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 476857 DEBUG (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 476857 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 476858 INFO  (TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testTriggerThrottling
   [junit4]   2> 476872 INFO  (TEST-TestSimTriggerIntegration.testCooldown-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCooldown
   [junit4]   2> 476895 DEBUG (TEST-TestSimTriggerIntegration.testCooldown-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10017_solr
   [junit4]   2> 476896 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 476896 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 476896 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 476896 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 476916 DEBUG (TEST-TestSimTriggerIntegration.testCooldown-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 476916 DEBUG (TEST-TestSimTriggerIntegration.testCooldown-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 476916 INFO  (TEST-TestSimTriggerIntegration.testCooldown-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 476945 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10017_solr]
   [junit4]   2> 476947 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 476947 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 476951 DEBUG (ScheduledTrigger-4662-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 476969 DEBUG (simCloudManagerPool-4661-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 476969 DEBUG (simCloudManagerPool-4661-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 476994 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 477008 DEBUG (ScheduledTrigger-4662-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 477008 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 477008 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 477012 DEBUG (simCloudManagerPool-4661-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 477012 DEBUG (simCloudManagerPool-4661-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 477012 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 477013 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 477013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger - Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10017_solr]
   [junit4]   2> 477013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 477013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 477013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 477013 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 477013 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 477028 DEBUG (ScheduledTrigger-4662-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 477033 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 477033 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 477044 DEBUG (simCloudManagerPool-4661-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 477044 DEBUG (simCloudManagerPool-4661-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 477044 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 477044 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 477044 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 477049 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 477050 DEBUG (TEST-TestSimTriggerIntegration.testCooldown-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- creating marker: /autoscaling/nodeAdded/127.0.0.1:10019_solr
   [junit4]   2> 477053 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 477053 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 477053 DEBUG (ScheduledTrigger-4662-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10019_solr at time 2537441907085662
   [junit4]   2> 477071 DEBUG (ScheduledTrigger-4662-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 477074 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 477074 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 477074 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10019_solr] added at times [2537441907085662], now=2537442949207262
   [junit4]   2> 477074 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"903ca3b82915eTb7796dwzlmgs1cgletigyjndt",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":2537441907085662,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537441907085662],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10019_solr"]}}
   [junit4]   2> 477075 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_cooldown_trigger]
   [junit4]   2> 477075 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903ca3b82915eTb7796dwzlmgs1cgletigyjndt",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":2537441907085662,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537441907085662],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537442964714262,
   [junit4]   2>     "nodeNames":["127.0.0.1:10019_solr"]}}
   [junit4]   2> 477079 DEBUG (ScheduledTrigger-4662-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10019_solr
   [junit4]   2> 477083 DEBUG (AutoscalingActionExecutor-4663-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"903ca3b82915eTb7796dwzlmgs1cgletigyjndt

[...truncated too long message...]

thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 479470 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 479470 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 479470 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing registered processor for nodes: [127.0.0.1:10043_solr] added at times [2537560220516512], now=2537562732312212
   [junit4]   2> 479470 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"903e5c78a40a0Tb7796dwzlmgs1cgletigyjnec",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":2537560220516512,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537560220516512],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10043_solr"]}}
   [junit4]   2> 479470 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_restore_trigger]
   [junit4]   2> 479470 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"903e5c78a40a0Tb7796dwzlmgs1cgletigyjnec",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":2537560220516512,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537560220516512],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537562740539212,
   [junit4]   2>     "nodeNames":["127.0.0.1:10043_solr"]}}
   [junit4]   2> 479471 DEBUG (ScheduledTrigger-4716-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10043_solr
   [junit4]   2> 479471 DEBUG (AutoscalingActionExecutor-4717-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"903e5c78a40a0Tb7796dwzlmgs1cgletigyjnec",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":2537560220516512,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537560220516512],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537562740539212,
   [junit4]   2>     "nodeNames":["127.0.0.1:10043_solr"]}}
   [junit4]   2> 479472 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 479472 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 479472 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 479472 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 479473 WARN  (AutoscalingActionExecutor-4717-thread-1) [    ] o.a.s.c.a.SystemLogListener Exception sending event. Collection: .system, event: {
   [junit4]   2>   "id":"903e5c78a40a0Tb7796dwzlmgs1cgletigyjnec",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":2537560220516512,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2537560220516512],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":2537562740539212,
   [junit4]   2>     "nodeNames":["127.0.0.1:10043_solr"]}}, exception: java.io.IOException: already closed
   [junit4]   2> 479473 WARN  (AutoscalingActionExecutor-4717-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Unhandled exception executing actions
   [junit4]   2> org.apache.solr.common.AlreadyClosedException: ScheduledTrigger node_added_restore_trigger has been closed.
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:583) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:334) [java/:?]
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [java/:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
   [junit4]   2> 479473 DEBUG (AutoscalingActionExecutor-4717-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 86 ms for event id=903e5c78a40a0Tb7796dwzlmgs1cgletigyjnec
   [junit4]   2> 479473 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 479473 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 479473 DEBUG (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 479474 INFO  (TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[FCBBB2AC9FCECAC8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_FCBBB2AC9FCECAC8-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4682c876), locale=cu-RU, timezone=America/Santa_Isabel
   [junit4]   2> NOTE: Linux 4.15.0-42-generic amd64/Oracle Corporation 12-ea (64-bit)/cpus=8,threads=1,free=181220096,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldCacheSortRandom, ZookeeperStatusHandlerTest, TestXmlQParser, HttpSolrCallGetCoreTest, TestCoreContainer, PingRequestHandlerTest, TestReplicationHandler, TestHdfsCloudBackupRestore, TestUninvertingReader, SimpleCollectionCreateDeleteTest, ShardsWhitelistTest, TestExceedMaxTermLength, DocumentBuilderTest, LeaderElectionContextKeyTest, TestLegacyBM25SimilarityFactory, TestLuceneIndexBackCompat, TestBulkSchemaAPI, ZkNodePropsTest, TestCloudManagedSchema, SolrCmdDistributorTest, PeerSyncWithLeaderTest, CdcrRequestHandlerTest, TestTolerantUpdateProcessorCloud, TestEmbeddedSolrServerSchemaAPI, TestDownShardTolerantSearch, TestPartialUpdateDeduplication, FieldAnalysisRequestHandlerTest, XmlInterpolationTest, TestDeleteCollectionOnDownNodes, TestConfigSetsAPIExclusivity, SimpleFacetsTest, TestXIncludeConfig, TestDistributedGrouping, TestStandardQParsers, DeleteNodeTest, TestScoreJoinQPNoScore, MetricsHandlerTest, AtomicUpdateProcessorFactoryTest, TestUnifiedSolrHighlighter, CollectionsAPIAsyncDistributedZkTest, BadIndexSchemaTest, MetricTriggerIntegrationTest, SolrShardReporterTest, SimplePostToolTest, SmileWriterTest, TestSolr4Spatial2, TestFunctionQuery, SynonymTokenizerTest, TestSerializedLuceneMatchVersion, TestRandomRequestDistribution, AutoScalingHandlerTest, TestCollationFieldDocValues, TestSolrDeletionPolicy2, TestDynamicFieldResource, TestCloudPseudoReturnFields, TestSimNodeAddedTrigger, TestWithCollection, SolrJmxReporterTest, TestFieldCache, OverseerTest, PeerSyncWithBufferUpdatesTest, DistributedQueryComponentOptimizationTest, SolrCoreTest, TestTestInjection, CustomCollectionTest, HdfsLockFactoryTest, NotRequiredUniqueKeyTest, TestSimTriggerIntegration]
   [junit4] Completed [206/839 (1!)] on J2 in 3.79s, 11 tests, 1 failure <<< FAILURES!

[...truncated 44648 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/36/consoleText

[repro] Revision: e9db95831b9db69fbc0bef499b0d3f41bc6448f1

[repro] Ant options: "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC"
[repro] Repro line:  ant test  -Dtestcase=TestSimTriggerIntegration -Dtests.method=testEventQueue -Dtests.seed=FCBBB2AC9FCECAC8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=cu-RU -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestSimTriggerIntegration
[repro] ant compile-test

[...truncated 3597 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestSimTriggerIntegration" -Dtests.showOutput=onerror "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC" -Dtests.seed=FCBBB2AC9FCECAC8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=cu-RU -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 125 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro] Exiting with code 0

[...truncated 40 lines...]

Mime
View raw message