lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS-EA] Lucene-Solr-7.x-Linux (64bit/jdk-12-ea+shipilev-fastdebug) - Build # 3593 - Unstable!
Date Fri, 01 Mar 2019 18:41:53 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3593/
Java: 64bit/jdk-12-ea+shipilev-fastdebug -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testSearchRate

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
	at __randomizedtesting.SeedInfo.seed([B7B7D87BD54DC7F8:EAFFC6F21A8B61B7]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testSearchRate(TestSimLargeCluster.java:682)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testSearchRate

Error Message:
The trigger did not start in time

Stack Trace:
java.lang.AssertionError: The trigger did not start in time
	at __randomizedtesting.SeedInfo.seed([B7B7D87BD54DC7F8:EAFFC6F21A8B61B7]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testSearchRate(TestSimTriggerIntegration.java:1370)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 13429 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimLargeCluster_B7B7D87BD54DC7F8-001/init-core-data-001
   [junit4]   2> 693408 INFO  (SUITE-TestSimLargeCluster-seed#[B7B7D87BD54DC7F8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 693408 INFO  (SUITE-TestSimLargeCluster-seed#[B7B7D87BD54DC7F8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 693408 INFO  (SUITE-TestSimLargeCluster-seed#[B7B7D87BD54DC7F8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 693411 INFO  (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 693414 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10004_solr
   [junit4]   2> 693414 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 693414 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 693414 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 693415 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 693415 INFO  (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 693415 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 693415 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 693415 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 693416 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 693416 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 693416 DEBUG (ScheduledTrigger-2679-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5 and last live nodes: 4
   [junit4]   2> 693426 DEBUG (ScheduledTrigger-2679-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 74 and last live nodes: 5
   [junit4]   2> 693450 DEBUG (ScheduledTrigger-2679-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 74
   [junit4]   2> 693456 DEBUG (simCloudManagerPool-2678-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 693457 DEBUG (simCloudManagerPool-2678-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 693457 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 693458 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 693458 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 693460 DEBUG (ScheduledTrigger-2679-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100
   [junit4]   2> 693463 DEBUG (simCloudManagerPool-2678-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 693464 DEBUG (simCloudManagerPool-2678-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 693464 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 693464 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, currentVersion=1
   [junit4]   2> 693464 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 693464 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 693464 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 693464 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 693470 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 693470 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 693476 DEBUG (simCloudManagerPool-2678-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 693476 DEBUG (simCloudManagerPool-2678-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 693482 DEBUG (simCloudManagerPool-2678-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 693482 DEBUG (simCloudManagerPool-2678-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 693488 DEBUG (simCloudManagerPool-2678-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=5
   [junit4]   2> 693489 DEBUG (simCloudManagerPool-2678-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 693489 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, currentVersion=6
   [junit4]   2> 693489 INFO  (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.CloudTestUtils -- shards without leaders: [shard1]
   [junit4]   2> 693489 DEBUG (simCloudManagerPool-2678-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 (currentVersion=6): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10036_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 693490 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=6
   [junit4]   2> 693491 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 693491 DEBUG (simCloudManagerPool-2678-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testSearchRate, currentVersion=7
   [junit4]   2> 693492 DEBUG (simCloudManagerPool-2678-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=7
   [junit4]   2> 693492 DEBUG (simCloudManagerPool-2678-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 693497 DEBUG (simCloudManagerPool-2678-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=8
   [junit4]   2> 693497 DEBUG (simCloudManagerPool-2678-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 693503 DEBUG (simCloudManagerPool-2678-thread-12) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=9
   [junit4]   2> 693504 DEBUG (simCloudManagerPool-2678-thread-12) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 9
   [junit4]   2> 693507 DEBUG (simCloudManagerPool-2678-thread-29) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testSearchRate / shard1 (currentVersion=10): {"core_node1":{
   [junit4]   2>     "core":"testSearchRate_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testSearchRate",
   [junit4]   2>     "node_name":"127.0.0.1:10056_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 693512 DEBUG (simCloudManagerPool-2678-thread-13) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=10
   [junit4]   2> 693513 DEBUG (simCloudManagerPool-2678-thread-13) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 10
   [junit4]   2> 693518 DEBUG (simCloudManagerPool-2678-thread-14) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=11
   [junit4]   2> 693519 DEBUG (simCloudManagerPool-2678-thread-14) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 11
   [junit4]   2> 693524 DEBUG (simCloudManagerPool-2678-thread-15) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=12
   [junit4]   2> 693525 DEBUG (simCloudManagerPool-2678-thread-15) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 12
   [junit4]   2> 693530 DEBUG (simCloudManagerPool-2678-thread-16) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=13
   [junit4]   2> 693531 DEBUG (simCloudManagerPool-2678-thread-16) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 13
   [junit4]   2> 693536 DEBUG (simCloudManagerPool-2678-thread-17) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=14
   [junit4]   2> 693537 DEBUG (simCloudManagerPool-2678-thread-17) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 14
   [junit4]   2> 693542 DEBUG (simCloudManagerPool-2678-thread-18) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=15
   [junit4]   2> 693543 DEBUG (simCloudManagerPool-2678-thread-18) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 15
   [junit4]   2> 693549 DEBUG (simCloudManagerPool-2678-thread-19) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=16
   [junit4]   2> 693549 DEBUG (simCloudManagerPool-2678-thread-19) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 16
   [junit4]   2> 693555 DEBUG (simCloudManagerPool-2678-thread-20) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=17
   [junit4]   2> 693555 DEBUG (simCloudManagerPool-2678-thread-20) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 17
   [junit4]   2> 693561 DEBUG (simCloudManagerPool-2678-thread-21) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=18
   [junit4]   2> 693561 DEBUG (simCloudManagerPool-2678-thread-21) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 18
   [junit4]   2> 693567 DEBUG (simCloudManagerPool-2678-thread-22) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=19
   [junit4]   2> 693568 DEBUG (simCloudManagerPool-2678-thread-22) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 19
   [junit4]   2> 693573 DEBUG (simCloudManagerPool-2678-thread-23) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=20
   [junit4]   2> 693574 DEBUG (simCloudManagerPool-2678-thread-23) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 20
   [junit4]   2> 693580 DEBUG (simCloudManagerPool-2678-thread-24) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=21
   [junit4]   2> 693581 DEBUG (simCloudManagerPool-2678-thread-24) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 21
   [junit4]   2> 693586 DEBUG (simCloudManagerPool-2678-thread-25) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=22
   [junit4]   2> 693587 DEBUG (simCloudManagerPool-2678-thread-25) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 22
   [junit4]   2> 693592 DEBUG (simCloudManagerPool-2678-thread-26) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=23
   [junit4]   2> 693593 DEBUG (simCloudManagerPool-2678-thread-26) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 23
   [junit4]   2> 693599 DEBUG (simCloudManagerPool-2678-thread-27) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=24
   [junit4]   2> 693599 DEBUG (simCloudManagerPool-2678-thread-27) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 24
   [junit4]   2> 693605 DEBUG (simCloudManagerPool-2678-thread-28) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=25
   [junit4]   2> 693605 DEBUG (simCloudManagerPool-2678-thread-28) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 25
   [junit4]   2> 693611 DEBUG (simCloudManagerPool-2678-thread-30) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=26
   [junit4]   2> 693612 DEBUG (simCloudManagerPool-2678-thread-30) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 26
   [junit4]   2> 693617 DEBUG (simCloudManagerPool-2678-thread-31) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=27
   [junit4]   2> 693618 DEBUG (simCloudManagerPool-2678-thread-31) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 27
   [junit4]   2> 693618 DEBUG (simCloudManagerPool-2678-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testSearchRate, currentVersion=28
   [junit4]   2> 693624 DEBUG (simCloudManagerPool-2678-thread-32) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=28
   [junit4]   2> 693624 DEBUG (simCloudManagerPool-2678-thread-32) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 28
   [junit4]   2> 693625 DEBUG (simCloudManagerPool-2678-thread-33) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testSearchRate / shard2 (currentVersion=29): {"core_node11":{
   [junit4]   2>     "core":"testSearchRate_shard2_replica_n11",
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testSearchRate",
   [junit4]   2>     "node_name":"127.0.0.1:10086_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 693626 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=29
   [junit4]   2> 693626 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 29
   [junit4]   2> 693626 INFO  (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.TestSimLargeCluster Ready after 769 ms
   [junit4]   2> 693626 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=30
   [junit4]   2> 693626 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 30
   [junit4]   2> 693746 DEBUG (simCloudManagerPool-2678-thread-43) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 693746 DEBUG (simCloudManagerPool-2678-thread-43) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 693746 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 693746 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 693747 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 693788 DEBUG (simCloudManagerPool-2678-thread-44) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 693788 DEBUG (simCloudManagerPool-2678-thread-44) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 693788 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 693789 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 693789 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 693916 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10004_solr
   [junit4]   2> 693953 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10035_solr
   [junit4]   2> 693953 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n3:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10022_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n4:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10017_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n7:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10081_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n9:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10036_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10071_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10007_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10070_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10093_solr
   [junit4]   2> 694007 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n10:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10067_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10086_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10083_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10057_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10090_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10087_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n5:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10030_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10011_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n2:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10069_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n6:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10075_solr
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n8:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694008 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10097_solr
   [junit4]   2> 694009 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10037_solr
   [junit4]   2> 694009 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10056_solr
   [junit4]   2> 694009 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n1:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 694027 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ## Collection: testSearchRate
   [junit4]   2> 694063 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##  - shard1
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n3  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n4  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n7  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n9  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n10  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n5  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n2  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n6  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n8  4.0
   [junit4]   2> 694098 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n1  4.0
   [junit4]   2> 695319 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger -- testSearchRate.shard1: totalShardRate=40.0, shardRate=4.0
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10004_solr
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10035_solr
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n3:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10022_solr
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n4:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10017_solr
   [junit4]   2> 695443 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n7:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10081_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n9:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10036_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10071_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10007_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10070_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10093_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n10:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10067_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10086_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10083_solr
   [junit4]   2> 695444 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10057_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10090_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10087_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n5:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10030_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10011_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n2:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10069_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n6:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10075_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n8:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10097_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10037_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ### rates for node 127.0.0.1:10056_solr
   [junit4]   2> 695445 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ###  metrics:solr.core.testSearchRate.shard1.replica_n1:QUERY./select.requestTimes:1minRate	4
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ## Collection: testSearchRate
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##  - shard1
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n3  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n4  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n7  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n9  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n10  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n5  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n2  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n6  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n8  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger ##     testSearchRate_shard1_replica_n1  4.0
   [junit4]   2> 695446 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SearchRateTrigger -- testSearchRate.shard1: totalShardRate=40.0, shardRate=4.0
   [junit4]   2> 696183 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3aeb742ae267aTacp9klnowf2in1up72pme726j",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":1036527051089530,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node3":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n3",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10035_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node4":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n4",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10022_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node7":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n7",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10017_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node9":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n9",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10081_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node10":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n10",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10093_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node5":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n5",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10087_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10011_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node6":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n6",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10069_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node8":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n8",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10075_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10056_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"testSearchRate":{"shard1":4.0}},
   [junit4]   2>     "hotCollections":{"testSearchRate":40.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10022_solr":4.0,
   [junit4]   2>       "127.0.0.1:10069_solr":4.0,
   [junit4]   2>       "127.0.0.1:10075_solr":4.0,
   [junit4]   2>       "127.0.0.1:10087_solr":4.0,
   [junit4]   2>       "127.0.0.1:10017_solr":4.0,
   [junit4]   2>       "127.0.0.1:10035_solr":4.0,
   [junit4]   2>       "127.0.0.1:10081_solr":4.0,
   [junit4]   2>       "127.0.0.1:10093_solr":4.0,
   [junit4]   2>       "127.0.0.1:10056_solr":4.0,
   [junit4]   2>       "127.0.0.1:10011_solr":4.0},
   [junit4]   2>     "requestedOps":[{
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"testSearchRate",
   [junit4]   2>               "second":"shard1"}]}}]}}
   [junit4]   2> 696211 DEBUG (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [search_rate_trigger]
   [junit4]   2> 696377 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 696378 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 696379 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 696379 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 696379 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 696379 WARN  (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SystemLogListener Exception sending event. Collection: .system, event: {
   [junit4]   2>   "id":"3aeb742ae267aTacp9klnowf2in1up72pme726j",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":1036527051089530,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node3":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n3",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10035_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node4":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n4",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10022_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node7":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n7",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10017_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node9":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n9",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10081_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node10":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n10",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10093_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node5":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n5",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10087_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10011_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node6":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n6",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10069_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node8":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n8",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10075_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10056_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"testSearchRate":{"shard1":4.0}},
   [junit4]   2>     "hotCollections":{"testSearchRate":40.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10022_solr":4.0,
   [junit4]   2>       "127.0.0.1:10069_solr":4.0,
   [junit4]   2>       "127.0.0.1:10075_solr":4.0,
   [junit4]   2>       "127.0.0.1:10087_solr":4.0,
   [junit4]   2>       "127.0.0.1:10017_solr":4.0,
   [junit4]   2>       "127.0.0.1:10035_solr":4.0,
   [junit4]   2>       "127.0.0.1:10081_solr":4.0,
   [junit4]   2>       "127.0.0.1:10093_solr":4.0,
   [junit4]   2>       "127.0.0.1:10056_solr":4.0,
   [junit4]   2>       "127.0.0.1:10011_solr":4.0},
   [junit4]   2>     "_enqueue_time_":1036738380203530,
   [junit4]   2>     "requestedOps":[{
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"testSearchRate",
   [junit4]   2>               "second":"shard1"}]}}]}}, exception: java.io.IOException: java.lang.InterruptedException
   [junit4]   2> 696381 WARN  (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.SystemLogListener Exception sending event. Collection: .system, event: {
   [junit4]   2>   "id":"3aeb742ae267aTacp9klnowf2in1up72pme726j",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":1036527051089530,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node3":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n3",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10035_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node4":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n4",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10022_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node7":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n7",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10017_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node9":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n9",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10081_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node10":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n10",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10093_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node5":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n5",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10087_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10011_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node6":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n6",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10069_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node8":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n8",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10075_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10056_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"testSearchRate":{"shard1":4.0}},
   [junit4]   2>     "hotCollections":{"testSearchRate":40.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10022_solr":4.0,
   [junit4]   2>       "127.0.0.1:10069_solr":4.0,
   [junit4]   2>       "127.0.0.1:10075_solr":4.0,
   [junit4]   2>       "127.0.0.1:10087_solr":4.0,
   [junit4]   2>       "127.0.0.1:10017_solr":4.0,
   [junit4]   2>       "127.0.0.1:10035_solr":4.0,
   [junit4]   2>       "127.0.0.1:10081_solr":4.0,
   [junit4]   2>       "127.0.0.1:10093_solr":4.0,
   [junit4]   2>       "127.0.0.1:10056_solr":4.0,
   [junit4]   2>       "127.0.0.1:10011_solr":4.0},
   [junit4]   2>     "_enqueue_time_":1036738380203530,
   [junit4]   2>     "requestedOps":[{
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"testSearchRate",
   [junit4]   2>               "second":"shard1"}]}}]}}, exception: java.io.IOException: already closed
   [junit4]   2> 696382 WARN  (ScheduledTrigger-2679-thread-4) [    ] o.a.s.c.a.ScheduledTriggers Ignoring autoscaling event {
   [junit4]   2>   "id":"3aeb742ae267aTacp9klnowf2in1up72pme726j",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":1036527051089530,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       {"core_node3":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n3",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10035_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node4":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n4",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10022_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node7":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n7",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10017_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node9":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n9",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10081_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node10":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n10",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10093_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node5":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n5",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10087_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node2":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n2",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10011_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node6":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n6",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10069_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node8":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n8",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10075_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       {"core_node1":{
   [junit4]   2>           "core":"testSearchRate_shard1_replica_n1",
   [junit4]   2>           "shard":"shard1",
   [junit4]   2>           "collection":"testSearchRate",
   [junit4]   2>           "node_name":"127.0.0.1:10056_solr",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "rate":4.0,
   [junit4]   2>           "state":"active",
   [junit4]   2>           "QUERY./select.requestTimes:1minRate":4,
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}}],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "violationType":["hotShards"],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"testSearchRate":{"shard1":4.0}},
   [junit4]   2>     "hotCollections":{"testSearchRate":40.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10022_solr":4.0,
   [junit4]   2>       "127.0.0.1:10069_solr":4.0,
   [junit4]   2>       "127.0.0.1:10075_solr":4.0,
   [junit4]   2>       "127.0.0.1:10087_solr":4.0,
   [junit4]   2>       "127.0.0.1:10017_solr":4.0,
   [junit4]   2>       "127.0.0.1:10035_solr":4.0,
   [junit4]   2>       "127.0.0.1:10081_solr":4.0,
   [junit4]   2>       "127.0.0.1:10093_solr":4.0,
   [junit4]   2>       "127.0.0.1:10056_solr":4.0,
   [junit4]   2>       "127.0.0.1:10011_solr":4.0},
   [junit4]   2>     "_enqueue_time_":1036738380203530,
   [junit4]   2>     "requestedOps":[{
   [junit4]   2>         "action":"ADDREPLICA",
   [junit4]   2>         "hints":{"COLL_SHARD":[{
   [junit4]   2>               "first":"testSearchRate",
   [junit4]   2>               "second":"shard1"}]}}]}} from trigger org.apache.solr.cloud.autoscaling.SearchRateTrigger@93cb0a70 because the executor has already been closed
   [junit4]   2> 696383 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 696383 DEBUG (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 696386 INFO  (TEST-TestSimLargeCluster.testSearchRate-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimLargeCluster -Dtests.method=testSearchRate -Dtests.seed=B7B7D87BD54DC7F8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-BM -Dtests.timezone=America/Caracas -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 3.05s J1 | TestSimLargeCluster.testSearchRate <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not fire at all
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B7B7D87BD54DC7F8:EAFFC6F21A8B61B7]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testSearchRate(TestSimLargeCluster.java:682)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:835)
   [junit4]   2> 696496 INFO  (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAddNode
   [junit4]   2> 696500 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10104_solr
   [junit4]   2> 696500 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 696501 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 696501 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 696501 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 696501 INFO  (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 696515 DEBUG (simCloudManagerPool-2684-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 696521 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 696521 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 696521 DEBUG (simCloudManagerPool-2684-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 696522 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10118_solr, 127.0.0.1:10196_solr, 127.0.0.1:10104_solr, 127.0.0.1:10182_solr, 127.0.0.1:10149_solr, 127.0.0.1:10179_solr, 127.0.0.1:10120_solr, 127.0.0.1:10165_solr, 127.0.0.1:10121_solr, 127.0.0.1:10166_solr, 127.0.0.1:10135_solr, 127.0.0.1:10151_solr, 127.0.0.1:10119_solr, 127.0.0.1:10178_solr, 127.0.0.1:10105_solr, 127.0.0.1:10150_solr, 127.0.0.1:10197_solr, 127.0.0.1:10148_solr, 127.0.0.1:10203_solr, 127.0.0.1:10137_solr, 127.0.0.1:10134_solr, 127.0.0.1:10194_solr, 127.0.0.1:10153_solr, 127.0.0.1:10183_solr, 127.0.0.1:10167_solr, 127.0.0.1:10180_solr, 127.0.0.1:10164_solr, 127.0.0.1:10199_solr, 127.0.0.1:10140_solr, 127.0.0.1:10146_solr, 127.0.0.1:10129_solr, 127.0.0.1:10123_solr, 127.0.0.1:10171_solr, 127.0.0.1:10107_solr, 127.0.0.1:10168_solr, 127.0.0.1:10124_solr, 127.0.0.1:10162_solr, 127.0.0.1:10177_solr, 127.0.0.1:10108_solr, 127.0.0.1:10193_solr, 127.0.0.1:10136_solr, 127.0.0.1:10139_solr, 127.0.0.1:10130_solr, 127.0.0.1:10133_solr, 127.0.0.1:10202_solr, 127.0.0.1:10155_solr, 127.0.0.1:10181_solr, 127.0.0.1:10152_solr, 127.0.0.1:10184_solr, 127.0.0.1:10187_solr, 127.0.0.1:10117_solr, 127.0.0.1:10114_solr, 127.0.0.1:10111_solr, 127.0.0.1:10112_solr, 127.0.0.1:10157_solr, 127.0.0.1:10190_solr, 127.0.0.1:10126_solr, 127.0.0.1:10160_solr, 127.0.0.1:10127_solr, 127.0.0.1:10173_solr, 127.0.0.1:10158_solr, 127.0.0.1:10113_solr, 127.0.0.1:10174_solr, 127.0.0.1:10143_solr, 127.0.0.1:10188_solr, 127.0.0.1:10161_solr, 127.0.0.1:10175_solr, 127.0.0.1:10186_solr, 127.0.0.1:10189_solr, 127.0.0.1:10145_solr, 127.0.0.1:10191_solr, 127.0.0.1:10200_solr, 127.0.0.1:10142_solr, 127.0.0.1:10159_solr, 127.0.0.1:10156_solr, 127.0.0.1:10172_solr, 127.0.0.1:10201_solr, 127.0.0.1:10154_solr, 127.0.0.1:10176_solr, 127.0.0.1:10131_solr, 127.0.0.1:10115_solr, 127.0.0.1:10169_solr, 127.0.0.1:10110_solr, 127.0.0.1:10163_solr, 127.0.0.1:10138_solr, 127.0.0.1:10170_solr, 127.0.0.1:10132_solr, 127.0.0.1:10116_solr, 127.0.0.1:10185_solr, 127.0.0.1:10122_solr, 127.0.0.1:10125_solr, 127.0.0.1:10144_solr, 127.0.0.1:10141_solr, 127.0.0.1:10192_solr, 127.0.0.1:10195_solr, 127.0.0.1:10128_solr, 127.0.0.1:10147_solr, 127.0.0.1:10106_solr, 127.0.0.1:10109_solr, 127.0.0.1:10198_solr]
   [junit4]   2> 696522 DEBUG (ScheduledTrigger-2685-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100
   [junit4]   2> 696522 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 696522 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 1
   [junit4]   2> 696522 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 696523 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 696523 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 696525 DEBUG (simCloudManagerPool-2684-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 696525 DEBUG (simCloudManagerPool-2684-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 696525 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 696525 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, currentVersion=1
   [junit4]   2> 696526 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 696526 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 696526 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 696526 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 696531 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 696531 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 696537 DEBUG (simCloudManagerPool-2684-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 696538 DEBUG (simCloudManagerPool-2684-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 696543 DEBUG (simCloudManagerPool-2684-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 696543 DEBUG (simCloudManagerPool-2684-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 696549 DEBUG (simCloudManagerPool-2684-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creati

[...truncated too long message...]

1728,
   [junit4]   2>     "nodeNames":["127.0.0.1:10032_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 2265053 DEBUG (ScheduledTrigger-9194-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 2265053 INFO  (AutoscalingActionExecutor-9195-thread-1) [    ] o.a.s.c.a.s.TestSimTriggerIntegration -- event: {
   [junit4]   2>   "id":"3b2511758ba68Tacp9klnowf2in1up72pme75jl",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":1040486283917928,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1040486283917928],
   [junit4]   2>     "_dequeue_time_":1040487706303378,
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":1040487320941728,
   [junit4]   2>     "nodeNames":["127.0.0.1:10032_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 2265053 DEBUG (ScheduledTrigger-9194-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 2265054 DEBUG (AutoscalingActionExecutor-9195-thread-1) [    ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
   [junit4]   2>   "id":"3b2511758ba68Tacp9klnowf2in1up72pme75jl",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":1040486283917928,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1040486283917928],
   [junit4]   2>     "_dequeue_time_":1040487706303378,
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":1040487320941728,
   [junit4]   2>     "nodeNames":["127.0.0.1:10032_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 2265054 DEBUG (AutoscalingActionExecutor-9195-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 2265055 DEBUG (AutoscalingActionExecutor-9195-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 2265055 DEBUG (AutoscalingActionExecutor-9195-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 78 ms for event id=3b2511758ba68Tacp9klnowf2in1up72pme75jl
   [junit4]   2> 2265150 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 2265150 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 2265151 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 2265151 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 2265151 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 2265151 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 2265151 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2265151 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[B7B7D87BD54DC7F8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_B7B7D87BD54DC7F8-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=sq-MK, timezone=Europe/Athens
   [junit4]   2> NOTE: Linux 4.18.0-15-generic amd64/Aleksey Shipilev 12-testing (64-bit)/cpus=8,threads=1,free=237430136,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [ZookeeperStatusHandlerTest, CopyFieldTest, NodeMutatorTest, NoCacheHeaderTest, TestCloudConsistency, TestTolerantUpdateProcessorRandomCloud, SolrCLIZkUtilsTest, XmlInterpolationTest, XsltUpdateRequestHandlerTest, TestSimDistributedQueue, MultiTermTest, ComputePlanActionTest, TestCollationField, ScriptEngineTest, SoftAutoCommitTest, TestReloadAndDeleteDocs, TestAddFieldRealTimeGet, TestSimGenericDistributedQueue, CoreAdminRequestStatusTest, TestSimpleQParserPlugin, HdfsWriteToMultipleCollectionsTest, SchemaApiFailureTest, CollectionsAPIDistributedZkTest, TestZkChroot, TestSimClusterStateProvider, TestDistribIDF, TestConfig, IndexSchemaRuntimeFieldTest, TestConfigSetProperties, PKIAuthenticationIntegrationTest, OpenCloseCoreStressTest, AddBlockUpdateTest, CloudExitableDirectoryReaderTest, SignificantTermsQParserPluginTest, MetricTriggerTest, TestFunctionQuery, TestNumericTerms32, MetricUtilsTest, TestLuceneMatchVersion, JavabinLoaderTest, TestFieldCacheReopen, CreateCollectionCleanupTest, TestQueryUtils, V2ApiIntegrationTest, TestSortByMinMaxFunction, TestJavabinTupleStreamParser, TestXmlQParser, TestConfigReload, SystemLogListenerTest, ShardsWhitelistTest, AnalysisAfterCoreReloadTest, TestOmitPositions, TestNonDefinedSimilarityFactory, TestLeaderInitiatedRecoveryThread, TriggerEventQueueTest, TestFieldTypeCollectionResource, HLLSerializationTest, TimeZoneUtilsTest, TestSubQueryTransformer, TestSweetSpotSimilarityFactory, TestPullReplicaErrorHandling, TestCollectionAPI, LIROnShardRestartTest, ShardRoutingTest, TestJmxIntegration, PreAnalyzedFieldTest, BlobRepositoryCloudTest, DistributedFacetPivotWhiteBoxTest, TestMacroExpander, TestHashQParserPlugin, InfixSuggestersTest, ZkStateReaderTest, ChaosMonkeyNothingIsSafeTest, DistributedVersionInfoTest, SolrJmxReporterTest, RangeFacetCloudTest, TestManagedResource, UnloadDistributedZkTest, QueryParsingTest, TestMaxScoreQueryParser, TestRebalanceLeaders, RemoteQueryErrorTest, ConcurrentCreateRoutedAliasTest, CdcrOpsAndBoundariesTest, TestDocSet, TestBackupRepositoryFactory, TestManagedSchema, DistributedFacetPivotLongTailTest, SearchHandlerTest, AliasIntegrationTest, PrimUtilsTest, TestUnifiedSolrHighlighter, RuleEngineTest, PeerSyncWithIndexFingerprintCachingTest, IndexBasedSpellCheckerTest, BasicDistributedZk2Test, FieldAnalysisRequestHandlerTest, Tagger2Test, UniqFieldsUpdateProcessorFactoryTest, TestReplicationHandler, TestHashPartitioner, TestDynamicFieldResource, LeaderFailureAfterFreshStartTest, TestDirectoryFactory, TestSchemaResource, SolrShardReporterTest, OverseerCollectionConfigSetProcessorTest, StatsReloadRaceTest, TestCoreDiscovery, TestSolrIndexConfig, SecurityConfHandlerTest, LargeFieldTest, TestLegacyTerms, TestFiltering, SpatialRPTFieldTypeTest, OpenExchangeRatesOrgProviderTest, TestLFUCache, StatsComponentTest, TestAnalyzedSuggestions, TestOnReconnectListenerSupport, TestExceedMaxTermLength, DistributedSuggestComponentTest, TestPivotHelperCode, MoveReplicaTest, TestSchemalessBufferedUpdates, ConjunctionSolrSpellCheckerTest, TermVectorComponentDistributedTest, TestClassicSimilarityFactory, GraphQueryTest, SpellPossibilityIteratorTest, TestDocTermOrds, ReplicationFactorTest, TestBulkSchemaConcurrent, BufferStoreTest, TestCloudRecovery, TestFastOutputStream, HdfsBasicDistributedZk2Test, TestImpersonationWithHadoopAuth, RestartWhileUpdatingTest, RollingRestartTest, SolrXmlInZkTest, SplitShardTest, TestCloudPhrasesIdentificationComponent, TestConfigSetsAPI, TestDistribDocBasedVersion, TestLeaderElectionZkExpiry, TestPullReplica, TestRandomFlRTGCloud, TestUtilizeNode, VMParamsZkACLAndCredentialsProvidersTest, ZkCLITest, AssignTest, CollectionReloadTest, CollectionsAPIAsyncDistributedZkTest, TestHdfsCloudBackupRestore, TestLocalFSCloudBackupRestore, AutoAddReplicasIntegrationTest, NodeMarkersRegistrationTest, ScheduledMaintenanceTriggerTest, ScheduledTriggerIntegrationTest, ScheduledTriggerTest, TestPolicyCloud, TestSimTriggerIntegration]
   [junit4] Completed [556/838 (2!)] on J0 in 4.01s, 11 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 43374 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3593/consoleText

[repro] Revision: 776013c52e58401c517b4bdd388a488520b84eb2

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseSerialGC"
[repro] Repro line:  ant test  -Dtestcase=TestSimLargeCluster -Dtests.method=testSearchRate -Dtests.seed=B7B7D87BD54DC7F8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-BM -Dtests.timezone=America/Caracas -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestSimTriggerIntegration -Dtests.method=testSearchRate -Dtests.seed=B7B7D87BD54DC7F8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sq-MK -Dtests.timezone=Europe/Athens -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestSimLargeCluster
[repro]       TestSimTriggerIntegration
[repro] ant compile-test

[...truncated 3588 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.TestSimLargeCluster|*.TestSimTriggerIntegration" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseSerialGC" -Dtests.seed=B7B7D87BD54DC7F8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-BM -Dtests.timezone=America/Caracas -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 165 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro] Exiting with code 0

[...truncated 43 lines...]

Mime
View raw message