lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_172) - Build # 23449 - Unstable!
Date Wed, 02 Jan 2019 18:10:02 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23449/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseG1GC

11 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode

Error Message:
no STARTED event

Stack Trace:
java.lang.AssertionError: no STARTED event
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:D4A3506647152101]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode(TestSimLargeCluster.java:318)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode

Error Message:
no STARTED event

Stack Trace:
java.lang.AssertionError: no STARTED event
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:D4A3506647152101]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode(TestSimLargeCluster.java:318)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode

Error Message:
no STARTED event

Stack Trace:
java.lang.AssertionError: no STARTED event
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:D4A3506647152101]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode(TestSimLargeCluster.java:318)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode

Error Message:
no STARTED event

Stack Trace:
java.lang.AssertionError: no STARTED event
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:D4A3506647152101]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode(TestSimLargeCluster.java:318)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode

Error Message:
no STARTED event

Stack Trace:
java.lang.AssertionError: no STARTED event
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:D4A3506647152101]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode(TestSimLargeCluster.java:318)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode

Error Message:
no STARTED event

Stack Trace:
java.lang.AssertionError: no STARTED event
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:D4A3506647152101]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testAddNode(TestSimLargeCluster.java:318)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0

Stack Trace:
java.lang.AssertionError: no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:CC59833B0BB2CB9F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.doTestNodeLost(TestSimLargeCluster.java:522)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost(TestSimLargeCluster.java:366)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0

Stack Trace:
java.lang.AssertionError: no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:CC59833B0BB2CB9F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.doTestNodeLost(TestSimLargeCluster.java:522)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost(TestSimLargeCluster.java:366)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0

Stack Trace:
java.lang.AssertionError: no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:CC59833B0BB2CB9F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.doTestNodeLost(TestSimLargeCluster.java:522)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost(TestSimLargeCluster.java:366)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0

Stack Trace:
java.lang.AssertionError: no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:CC59833B0BB2CB9F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.doTestNodeLost(TestSimLargeCluster.java:522)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost(TestSimLargeCluster.java:366)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0

Stack Trace:
java.lang.AssertionError: no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0
	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:CC59833B0BB2CB9F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.doTestNodeLost(TestSimLargeCluster.java:522)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost(TestSimLargeCluster.java:366)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14534 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimLargeCluster_734C4DC58858AE19-001/init-core-data-001
   [junit4]   2> 1793982 INFO  (SUITE-TestSimLargeCluster-seed#[734C4DC58858AE19]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1793983 INFO  (SUITE-TestSimLargeCluster-seed#[734C4DC58858AE19]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1793983 INFO  (SUITE-TestSimLargeCluster-seed#[734C4DC58858AE19]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1793986 INFO  (TEST-TestSimLargeCluster.testAddNode-seed#[734C4DC58858AE19]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAddNode
   [junit4]   2> 1793993 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10044_solr
   [junit4]   2> 1793993 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 1793993 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 1793993 DEBUG (TEST-TestSimLargeCluster.testAddNode-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 1793993 INFO  (TEST-TestSimLargeCluster.testAddNode-seed#[734C4DC58858AE19]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1793993 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1793994 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 1793994 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1793994 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10049_solr, 127.0.0.1:10050_solr, 127.0.0.1:10054_solr, 127.0.0.1:10046_solr, 127.0.0.1:10047_solr, 127.0.0.1:10044_solr, 127.0.0.1:10052_solr, 127.0.0.1:10055_solr, 127.0.0.1:10045_solr, 127.0.0.1:10051_solr, 127.0.0.1:10053_solr, 127.0.0.1:10048_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 1793994 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 1793994 DEBUG (ScheduledTrigger-8969-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 15 and last live nodes: 13
   [junit4]   2> 1793994 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1794003 DEBUG (simCloudManagerPool-8968-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1794003 DEBUG (simCloudManagerPool-8968-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1794004 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1794007 DEBUG (simCloudManagerPool-8968-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1794007 DEBUG (simCloudManagerPool-8968-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1794007 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 1794008 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 2
   [junit4]   2> 1794008 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1794008 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 1794008 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1794008 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 15
   [junit4]   2> 1794012 DEBUG (simCloudManagerPool-8968-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1794013 DEBUG (simCloudManagerPool-8968-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1794013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1794013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 - Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10135_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10134_solr, 127.0.0.1:10137_solr, 127.0.0.1:10098_solr, 127.0.0.1:10140_solr, 127.0.0.1:10054_solr, 127.0.0.1:10129_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10136_solr, 127.0.0.1:10139_solr, 127.0.0.1:10067_solr, 127.0.0.1:10061_solr, 127.0.0.1:10130_solr, 127.0.0.1:10133_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10117_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10126_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10127_solr, 127.0.0.1:10113_solr, 127.0.0.1:10058_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10143_solr, 127.0.0.1:10044_solr, 127.0.0.1:10092_solr, 127.0.0.1:10087_solr, 127.0.0.1:10142_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10131_solr, 127.0.0.1:10115_solr, 127.0.0.1:10110_solr, 127.0.0.1:10069_solr, 127.0.0.1:10138_solr, 127.0.0.1:10062_solr, 127.0.0.1:10132_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10125_solr, 127.0.0.1:10075_solr, 127.0.0.1:10050_solr, 127.0.0.1:10078_solr, 127.0.0.1:10141_solr, 127.0.0.1:10128_solr, 127.0.0.1:10097_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 1794013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=start, class=org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster$StartTriggerAction}, {name=compute, class=org.apache.solr.cloud.autoscaling.ComputePlanAction}, {name=execute, class=org.apache.solr.cloud.autoscaling.ExecutePlanAction}, {name=test, class=org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster$FinishTriggerAction}], enabled=true}
   [junit4]   2> 1794013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 1794013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1794013 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794013 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794014 DEBUG (simCloudManagerPool-8968-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testNodeAdded, currentVersion=1
   [junit4]   2> 1794014 DEBUG (simCloudManagerPool-8968-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 1794014 DEBUG (simCloudManagerPool-8968-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 1794021 DEBUG (simCloudManagerPool-8968-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 1794021 DEBUG (simCloudManagerPool-8968-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 1794023 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794024 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794034 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794034 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794044 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794044 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794054 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794054 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794064 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794064 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794075 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794075 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794085 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794085 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794095 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794095 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794105 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794105 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794116 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794116 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794126 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794126 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794136 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794136 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794166 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794166 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794176 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794176 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794186 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794186 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794197 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794197 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794207 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794207 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794217 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794217 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794227 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794227 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794238 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794238 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794248 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794248 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794258 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794258 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794268 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794268 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794278 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794279 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794289 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794289 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794299 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794299 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794310 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794310 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794320 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794320 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794331 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794331 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794341 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794341 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794351 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794351 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794361 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794361 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794371 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794372 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794382 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794382 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794392 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794392 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794402 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794402 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794412 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794412 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794423 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794423 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794439 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794439 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794449 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794449 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794459 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794460 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794470 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794470 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794480 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794480 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794490 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794490 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794500 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794500 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794511 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794511 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794521 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794521 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794531 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794531 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794541 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794541 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794552 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794552 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794562 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794562 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794572 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794572 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794582 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794582 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794592 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794593 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794603 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794603 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794613 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794613 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794623 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794623 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794633 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794633 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794644 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794644 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794654 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794654 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794664 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794664 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794683 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794683 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794693 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794693 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794703 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794703 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794713 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794713 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794724 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794724 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794734 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794734 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794744 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794744 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794754 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794754 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794765 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794765 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794775 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794775 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794785 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794785 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794795 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794795 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794806 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794806 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794816 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794816 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794826 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794826 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794836 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794836 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794847 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794847 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794857 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794857 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794867 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794867 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794877 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794877 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794887 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794887 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794898 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794898 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794908 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794908 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794918 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794918 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794928 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794928 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794939 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794939 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794954 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794954 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794964 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794964 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794975 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794975 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794985 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794985 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1794995 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1794995 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795005 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795005 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795015 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795016 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795026 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795026 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795036 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795036 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795046 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795046 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795056 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795056 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795067 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795067 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795077 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795077 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795087 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795087 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795097 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795097 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795107 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795108 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795118 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795118 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795128 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795128 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795138 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795138 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795148 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795149 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795159 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795159 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795169 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795169 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795179 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795179 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795190 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795190 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795200 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795200 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795210 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795210 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795220 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795220 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795235 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795235 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795245 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795245 DEBUG (ScheduledTrigger-8969-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795255 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795256 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795266 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795266 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795276 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795276 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795286 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795286 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795296 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795296 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795307 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795307 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795317 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795317 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795327 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1795327 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 1795337 DEBUG (ScheduledTrigger-8969-thread-3) [    ] o.a.s.c.a.NodeAddedTr

[...truncated too long message...]

rThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 52645 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 52646 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 52646 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 52686 DEBUG (AutoscalingActionExecutor-34-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testNodeLost&targetNode=127.0.0.1:10560_solr&inPlaceMove=true&replica=core_node94
   [junit4]   2> 52686 WARN  (AutoscalingActionExecutor-34-thread-1) [    ] o.a.s.c.a.SystemLogListener Exception sending event. Collection: .system, event: {
   [junit4]   2>   "id":"4f3c55ba2a0eeTe90yuan7yxi96ka3mf8vrdqhb",
   [junit4]   2>   "source":"node_lost_trigger3",
   [junit4]   2>   "eventTime":1393928878334190,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       1393928878334190,
   [junit4]   2>       1393928874430890,
   [junit4]   2>       1393928876058790,
   [junit4]   2>       1393928876856490,
   [junit4]   2>       1393928877579890],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":1393967030338890,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10549_solr",
   [junit4]   2>       "127.0.0.1:10620_solr",
   [junit4]   2>       "127.0.0.1:10634_solr",
   [junit4]   2>       "127.0.0.1:10618_solr",
   [junit4]   2>       "127.0.0.1:10581_solr"]}}, exception: java.io.IOException: already closed
   [junit4]   2> 52686 WARN  (AutoscalingActionExecutor-34-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.solr.cloud.autoscaling.TriggerActionException: Error processing action for trigger event: {
   [junit4]   2>   "id":"4f3c55ba2a0eeTe90yuan7yxi96ka3mf8vrdqhb",
   [junit4]   2>   "source":"node_lost_trigger3",
   [junit4]   2>   "eventTime":1393928878334190,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       1393928878334190,
   [junit4]   2>       1393928874430890,
   [junit4]   2>       1393928876058790,
   [junit4]   2>       1393928876856490,
   [junit4]   2>       1393928877579890],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":1393967030338890,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10549_solr",
   [junit4]   2>       "127.0.0.1:10620_solr",
   [junit4]   2>       "127.0.0.1:10634_solr",
   [junit4]   2>       "127.0.0.1:10618_solr",
   [junit4]   2>       "127.0.0.1:10581_solr"]}}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:327) ~[java/:?]
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception while processing event: {
   [junit4]   2>   "id":"4f3c55ba2a0eeTe90yuan7yxi96ka3mf8vrdqhb",
   [junit4]   2>   "source":"node_lost_trigger3",
   [junit4]   2>   "eventTime":1393928878334190,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       1393928878334190,
   [junit4]   2>       1393928874430890,
   [junit4]   2>       1393928876058790,
   [junit4]   2>       1393928876856490,
   [junit4]   2>       1393928877579890],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":1393967030338890,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10549_solr",
   [junit4]   2>       "127.0.0.1:10620_solr",
   [junit4]   2>       "127.0.0.1:10634_solr",
   [junit4]   2>       "127.0.0.1:10618_solr",
   [junit4]   2>       "127.0.0.1:10581_solr"]}}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:160) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324) ~[java/:?]
   [junit4]   2> 	... 6 more
   [junit4]   2> Caused by: java.lang.InterruptedException: stopping - thread was interrupted
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:112) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324) ~[java/:?]
   [junit4]   2> 	... 6 more
   [junit4]   2> 52687 DEBUG (AutoscalingActionExecutor-34-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 40197 ms for event id=4f3c55ba2a0eeTe90yuan7yxi96ka3mf8vrdqhb
   [junit4]   2> 52687 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 52687 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 52687 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 52705 INFO  (TEST-TestSimLargeCluster.testNodeLost-seed#[734C4DC58858AE19]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLost
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimLargeCluster -Dtests.method=testNodeLost -Dtests.seed=734C4DC58858AE19 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=it-CH -Dtests.timezone=America/Scoresbysund -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 8.07s J0 | TestSimLargeCluster.testNodeLost <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: no STARTED event: [], waitFor=5, killDelay=5000, minIgnored=0
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([734C4DC58858AE19:CC59833B0BB2CB9F]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.doTestNodeLost(TestSimLargeCluster.java:522)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost(TestSimLargeCluster.java:366)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimLargeCluster_734C4DC58858AE19-002
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1161, maxMBSortInHeap=6.896133156210676, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@114b8c4), locale=it-CH, timezone=America/Scoresbysund
   [junit4]   2> NOTE: Linux 4.15.0-42-generic i386/Oracle Corporation 1.8.0_172 (32-bit)/cpus=8,threads=1,free=397495312,total=449839104
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimLargeCluster, TestSimLargeCluster]
   [junit4] Completed [5/5 (5!)] on J0 in 21.30s, 4 tests, 2 failures, 1 skipped <<< FAILURES!

[...truncated 20 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1572: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1099: There were test failures: 5 suites, 20 tests, 10 failures, 5 ignored (5 assumptions) [seed: 734C4DC58858AE19]

Total time: 56 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=a1c6e642aad90d3615b4c71bf261a5aad7e32369, workspace=/var/lib/jenkins/workspace/Lucene-Solr-master-Linux
[WARNINGS] Computing warning deltas based on reference build #23448
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

Mime
View raw message