lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_172) - Build # 7544 - Unstable!
Date Mon, 01 Oct 2018 02:31:52 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7544/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue {#2}

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue

Error Message:
action did not start

Stack Trace:
java.lang.AssertionError: action did not start
	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3:687960581151CE46]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:666)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14030 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestSimTriggerIntegration_A1CC22F6183608B3-001\init-core-data-001
   [junit4]   2> 5617050 INFO  (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 5617050 INFO  (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 5617050 INFO  (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 5617059 DEBUG (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10324_solr
   [junit4]   2> 5617060 DEBUG (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617060 DEBUG (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version -1
   [junit4]   2> 5617060 INFO  (SUITE-TestSimTriggerIntegration-seed#[A1CC22F6183608B3]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 5617060 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 5617060 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 5617060 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 5617060 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 5617061 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5617061 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617061 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 5617065 DEBUG (ScheduledTrigger-17738-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617075 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 5617075 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 5617075 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 5617077 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 5617079 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 5617082 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617082 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 5617083 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 5617084 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 5617084 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 5617084 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 5617084 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 5617085 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5617085 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617085 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 5617085 DEBUG (ScheduledTrigger-17743-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617106 DEBUG (ScheduledTrigger-17743-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617130 DEBUG (ScheduledTrigger-17743-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617151 DEBUG (ScheduledTrigger-17743-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617172 DEBUG (ScheduledTrigger-17743-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617185 DEBUG (simCloudManagerPool-17742-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617187 DEBUG (simCloudManagerPool-17742-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 5617187 DEBUG (simCloudManagerPool-17742-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5617187 DEBUG (simCloudManagerPool-17742-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 5617189 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 5617189 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617189 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 5617189 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617189 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 5617192 DEBUG (ScheduledTrigger-17743-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617211 DEBUG (simCloudManagerPool-17742-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617211 DEBUG (simCloudManagerPool-17742-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 5617211 DEBUG (simCloudManagerPool-17742-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10324_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 5617213 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617213 DEBUG (ScheduledTrigger-17743-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617213 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 5617215 DEBUG (simCloudManagerPool-17742-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5617216 DEBUG (simCloudManagerPool-17742-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 5617216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 5617216 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 5617216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5617216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10326_solr
   [junit4]   2> 5617216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 5617216 DEBUG (ScheduledTrigger-17743-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
   [junit4]   2> 5617233 DEBUG (ScheduledTrigger-17743-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617240 DEBUG (ScheduledTrigger-17743-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
   [junit4]   2> 5617248 DEBUG (simCloudManagerPool-17742-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5617249 DEBUG (simCloudManagerPool-17742-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 5617249 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 5617249 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 5617249 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5617249 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617249 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 5617249 DEBUG (ScheduledTrigger-17743-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
   [junit4]   2> 5617249 DEBUG (ScheduledTrigger-17743-thread-4) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10326_solr]
   [junit4]   2> 5617254 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 5617254 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617254 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 5617254 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		2
   [junit4]   2> ## Empty nodes:	1
   [junit4]   2> ## Dead nodes:		1
   [junit4]   2> ##		127.0.0.1:10326_solr
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * .system
   [junit4]   2> ##    shardsTotal	1
   [junit4]   2> ##    shardsState	{active=1}
   [junit4]   2> ##      shardsWithoutLeader	0
   [junit4]   2> ##    totalReplicas	1
   [junit4]   2> ##      activeReplicas	1
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	2
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	2
   [junit4]   2> ##      minActiveSliceDocs	2
   [junit4]   2> ##      avgActiveSliceDocs	2
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	11,240
   [junit4]   2> ##      maxActiveSliceBytes	11,240
   [junit4]   2> ##      minActiveSliceBytes	11,240
   [junit4]   2> ##      avgActiveSliceBytes	11,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	1
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		1
   [junit4]   2> ## * .system		1
   [junit4]   2> ##		- active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- autoscaling        3
   [junit4]   2> ##		- update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_restore_trigger
   [junit4]   2> ##		- STARTED         1
   [junit4]   2> ##		- SUCCEEDED       1
   [junit4]   2> 
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeMarkersRegistration
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 5617291 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 5617292 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 5617292 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 5617292 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10326_solr
   [junit4]   2> 5617292 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 5617294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 5617294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617294 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 5617294 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 5617295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 5617295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 5617295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 5617295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 5617296 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5617296 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617296 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 5617296 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617317 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617339 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617360 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617381 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617397 DEBUG (simCloudManagerPool-17745-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617397 DEBUG (simCloudManagerPool-17745-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 5617397 DEBUG (simCloudManagerPool-17745-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5617397 DEBUG (simCloudManagerPool-17745-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 5617399 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 5617399 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617399 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5617399 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 5617399 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 5617405 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5617419 DEBUG (simCloudManagerPool-17745-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617419 DEBUG (simCloudManagerPool-17745-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 5617419 DEBUG (simCloudManagerPool-17745-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: {"core_node2":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10324_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 5617421 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5617421 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 5617423 INFO  (simCloudManagerPool-17745-thread-4) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 5617423 DEBUG (simCloudManagerPool-17745-thread-4) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 5617423 DEBUG (simCloudManagerPool-17745-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5617423 DEBUG (simCloudManagerPool-17745-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 5617423 INFO  (simCloudManagerPool-17745-thread-4) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 5617423 DEBUG (simCloudManagerPool-17745-thread-4) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 5617423 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 5617425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr, 127.0.0.1:10327_solr]
   [junit4]   2> 5617425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestEventQueueAction}], enabled=true}
   [junit4]   2> 5617425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 5617425 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617425 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617425 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617446 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617449 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617449 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617470 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617470 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617470 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617492 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617492 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617492 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617512 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617512 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617512 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617533 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617533 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617533 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617555 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617555 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617555 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617575 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617575 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617575 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617596 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617596 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617596 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617618 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617618 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617618 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617639 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617639 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617639 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617660 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617660 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617660 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617680 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617680 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617680 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617701 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617701 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617701 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617723 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617723 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617723 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617744 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617744 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617744 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617764 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617764 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617764 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617785 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617785 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617785 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617806 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617806 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617806 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617827 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617827 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617827 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617847 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617847 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617847 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617868 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617868 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617868 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617890 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617890 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617890 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617910 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617910 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617910 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617931 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617931 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617931 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617952 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617952 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617952 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617973 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617973 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617973 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5617994 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5617994 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5617994 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618014 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618014 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618014 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618035 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618035 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618035 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618056 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618056 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618056 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618076 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618076 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618076 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618097 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618097 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618097 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618117 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618117 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618117 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618138 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618138 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618138 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618158 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618158 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618158 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618182 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618182 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618182 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618202 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618202 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618202 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618222 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618222 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618222 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618243 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618243 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618243 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618263 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618263 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618263 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618284 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618284 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618284 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618304 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618304 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618304 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618325 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618325 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618325 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618345 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618345 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618345 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618365 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618365 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618365 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618386 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618386 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618386 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618406 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618406 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618406 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618427 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618427 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618427 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618448 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618448 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618448 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618468 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618468 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618468 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618489 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618489 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618489 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618510 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618510 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618510 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618531 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618531 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618531 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618552 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618552 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618552 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618574 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618574 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618574 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618594 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618594 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618594 DEBUG (ScheduledTrigger-17746-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618615 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618615 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618615 DEBUG (ScheduledTrigger-17746-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618627 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> 5618627 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618627 DEBUG (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 9
   [junit4]   2> 5618627 INFO  (TEST-TestSimTriggerIntegration.testEventQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		3
   [junit4]   2> ## Empty nodes:	2
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * .system
   [junit4]   2> ##    shardsTotal	1
   [junit4]   2> ##    shardsState	{active=1}
   [junit4]   2> ##      shardsWithoutLeader	0
   [junit4]   2> ##    totalReplicas	1
   [junit4]   2> ##      activeReplicas	1
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	0
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	0
   [junit4]   2> ##      minActiveSliceDocs	0
   [junit4]   2> ##      avgActiveSliceDocs	0
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	10,240
   [junit4]   2> ##      maxActiveSliceBytes	10,240
   [junit4]   2> ##      minActiveSliceBytes	10,240
   [junit4]   2> ##      avgActiveSliceBytes	10,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	1
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		1
   [junit4]   2> ## * .system		1
   [junit4]   2> ##		- active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- autoscaling        2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_restore_trigger
   [junit4]   2> ##		- STARTED         1
   [junit4]   2> ##		- SUCCEEDED       1
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimTriggerIntegration -Dtests.method=testEventQueue -Dtests.seed=A1CC22F6183608B3 -Dtests.slow=true -Dtests.locale=en-NZ -Dtests.timezone=Mexico/General -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.38s J0 | TestSimTriggerIntegration.testEventQueue <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: action did not start
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3:687960581151CE46]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:666)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 5618636 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 5618636 DEBUG (ScheduledTrigger-17746-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5618636 DEBUG (ScheduledTrigger-17746-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5618654 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
   [junit4]   2> 5618654 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 5618654 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 5618654 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10327_solr
   [junit4]   2> 5618654 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 5618654 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 5618654 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618654 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 5618657 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 5618658 DEBUG (ScheduledTrigger-17749-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618680 DEBUG (ScheduledTrigger-17749-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618704 DEBUG (ScheduledTrigger-17749-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618724 DEBUG (ScheduledTrigger-17749-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618746 DEBUG (ScheduledTrigger-17749-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618758 DEBUG (simCloudManagerPool-17748-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618758 DEBUG (simCloudManagerPool-17748-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 10
   [junit4]   2> 5618759 DEBUG (simCloudManagerPool-17748-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5618759 DEBUG (simCloudManagerPool-17748-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
   [junit4]   2> 5618761 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
   [junit4]   2> 5618763 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618763 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 5618763 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618763 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 11
   [junit4]   2> 5618766 DEBUG (ScheduledTrigger-17749-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618784 DEBUG (simCloudManagerPool-17748-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618784 DEBUG (simCloudManagerPool-17748-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 12
   [junit4]   2> 5618784 DEBUG (simCloudManagerPool-17748-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: {"core_node3":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10325_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 5618786 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618786 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 13
   [junit4]   2> 5618787 DEBUG (simCloudManagerPool-17748-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5618787 DEBUG (simCloudManagerPool-17748-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
   [junit4]   2> 5618787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 5618787 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 5618787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10325_solr]
   [junit4]   2> 5618787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 5618787 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618787 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 14
   [junit4]   2> 5618788 DEBUG (ScheduledTrigger-17749-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 5618788 DEBUG (ScheduledTrigger-17749-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 5618788 DEBUG (ScheduledTrigger-17749-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10325_solr
   [junit4]   2> 5618788 DEBUG (ScheduledTrigger-17749-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10325_solr
   [junit4]   2> 5618789 WARN  (simCloudManagerPool-17748-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider Can't find any active replicas for .system / shard1: shard1:{
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10325_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 5618789 DEBUG (simCloudManagerPool-17748-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider -- liveNodes: [127.0.0.1:10324_solr]
   [junit4]   2> 5618810 DEBUG (ScheduledTrigger-17749-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 5618810 DEBUG (ScheduledTrigger-17749-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 5618830 DEBUG (ScheduledTrigger-17749-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 5618830 DEBUG (ScheduledTrigger-17749-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 5618830 DEBUG (ScheduledTrigger-17749-thread-1) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10325_solr]
   [junit4]   2> 5618832 DEBUG (simCloudManagerPool-17748-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10325_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 5618833 DEBUG (simCloudManagerPool-17748-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10325_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 5618834 DEBUG (simCloudManagerPool-17748-thread-7) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5618834 DEBUG (simCloudManagerPool-17748-thread-7) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
   [junit4]   2> 5618834 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
   [junit4]   2> 5618834 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618834 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 5618877 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
   [junit4]   2> 5618877 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5618877 DEBUG (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 15
   [junit4]   2> 5618877 INFO  (TEST-TestSimTriggerIntegration.testNodeLostTrigger-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		1
   [junit4]   2> ## Empty nodes:	1
   [junit4]   2> ## Dead nodes:		1
   [junit4]   2> ##		127.0.0.1:10325_solr
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * .system
   [junit4]   2> ##    shardsTotal	1
   [junit4]   2> ##    shardsState	{active=1}
   [junit4]   2> ##      shardsWithoutLeader	1
   [junit4]   2> ##    totalReplicas	1
   [junit4]   2> ##      activeReplicas	1
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	0
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	0
   [junit4]   2> ##      minActiveSliceDocs	0
   [junit4]   2> ##      avgActiveSliceDocs	0
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	10,240
   [junit4]   2> ##      maxActiveSliceBytes	10,240
   [junit4]   2> ##      minActiveSliceBytes	10,240
   [junit4]   2> ##      avgActiveSliceBytes	10,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	0
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- autoscaling        3
   [junit4]   2> ##		- update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_restore_trigger
   [junit4]   2> ##		- STARTED         1
   [junit4]   2> ##		- SUCCEEDED       1
   [junit4]   2> ## * Trigger: node_lost_trigger
   [junit4]   2> ##		- STARTED         1
   [junit4]   2> ##		- SUCCEEDED       1
   [junit4]   2> 
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testCooldown
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 5618919 INFO  (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
   [junit4]   2> 5618919 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
   [junit4]   2> 5618919 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
   [junit4]   2> 5618919 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10325_solr
   [junit4]   2> 5618919 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 5618919 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618919 INFO  (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 5618919 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
   [junit4]   2> 5618919 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 5618920 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 5618921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 5618921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
   [junit4]   2> 5618922 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion -1
   [junit4]   2> 5618922 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
   [junit4]   2> 5618922 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10328_solr]
   [junit4]   2> 5618922 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5618922 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 5618922 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618944 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618964 DEBUG (ScheduledTrigger-17752-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5618985 DEBUG (ScheduledTrigger-17752-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5619006 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5619021 DEBUG (simCloudManagerPool-17751-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5619021 DEBUG (simCloudManagerPool-17751-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 16
   [junit4]   2> 5619022 DEBUG (simCloudManagerPool-17751-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5619022 DEBUG (simCloudManagerPool-17751-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
   [junit4]   2> 5619024 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
   [junit4]   2> 5619024 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5619026 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5619026 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 17
   [junit4]   2> 5619026 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 5619026 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
   [junit4]   2> 5619048 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 5619048 DEBUG (simCloudManagerPool-17751-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5619048 DEBUG (simCloudManagerPool-17751-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 18
   [junit4]   2> 5619048 DEBUG (simCloudManagerPool-17751-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: {"core_node4":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10324_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 5619050 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [junit4]   2> 5619050 DEBUG (TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A1CC22F6183608B3]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 19
   [junit4]   2> 5619051 DEBUG (simCloudManagerPool-17751-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 5619051 DEBUG (simCloudManagerPool-17751-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
   [junit4]   2> 5619051 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 18
   [junit4]   2> 5619051 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 5619051 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10324_solr, 127.0.0.1:10328_solr]
   [junit4]   2> 5619051 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 5619051 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion 18
   [junit4]   2> 5619051 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619052 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619052 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10329_solr at time 1464143410493000
   [junit4]   2> 5619068 DEBUG (ScheduledTrigger-17752-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619074 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619074 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619089 DEBUG (ScheduledTrigger-17752-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619095 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619095 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619110 DEBUG (ScheduledTrigger-17752-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619116 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619116 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619130 DEBUG (ScheduledTrigger-17752-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619136 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619136 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619153 DEBUG (ScheduledTrigger-17752-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619156 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619156 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619174 DEBUG (ScheduledTrigger-17752-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619177 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619177 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619194 DEBUG (ScheduledTrigger-17752-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619198 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619198 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619215 DEBUG (ScheduledTrigger-17752-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619218 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619218 DEBUG (ScheduledTrigger-17752-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619236 DEBUG (ScheduledTrigger-17752-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619239 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 5619239 DEBUG (ScheduledTrigger-17752-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 5619259 DEBUG (ScheduledTrigger-17752-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 5619260 DEBUG (ScheduledTrigger-17752-thread

[...truncated too long message...]

.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueue(TestSimDistributedQueue.java:74)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue(TestSimGenericDistributedQueue.java:36)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> ??? 01, 2018 8:28:46 ?? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=13, name=TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[A1CC22F6183608B3], state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]   2> 7284910 INFO  (TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[A1CC22F6183608B3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_A1CC22F6183608B3-001
   [junit4]   2> Oct 01, 2018 2:28:46 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=917, maxMBSortInHeap=6.19659064061721, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1259e48), locale=el-GR, timezone=BST
   [junit4]   2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_172 (32-bit)/cpus=3,threads=1,free=48961800,total=64880640
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimGenericDistributedQueue]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimGenericDistributedQueue -Dtests.seed=A1CC22F6183608B3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=el-GR -Dtests.timezone=BST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestSimGenericDistributedQueue (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A1CC22F6183608B3]:0)
   [junit4] Completed [9/10 (1!)] on J0 in 7283.46s, 1 test, 2 errors <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:1567: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:1092: There were test failures: 10 suites, 76 tests, 1 suite-level error, 1 error [seed: A1CC22F6183608B3]

Total time: 121 minutes 35 seconds

[repro] Setting last failure code to 1

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro]   1/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
[repro] Exiting with code 1

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows>IF EXIST lucene\build move /y lucene\build lucene\build.repro 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows>IF EXIST solr\build move /y solr\build solr\build.repro 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows>IF EXIST lucene\build.orig move /y lucene\build.orig lucene\build 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows>IF EXIST solr\build.orig move /y solr\build.orig solr\build 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows>exit 0 
Archiving artifacts
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=964cc88cee7d62edf03a923e3217809d630af5d5, workspace=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows
[WARNINGS] Computing warning deltas based on reference build #7543
Recording test results
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2

Mime
View raw message