lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 820 - Still Failing
Date Tue, 13 Oct 2015 13:58:00 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/820/

4 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 30 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 30 seconds and no jetties were stopped - this is worth investigating!
	at __randomizedtesting.SeedInfo.seed([79703BC618647C1C:F124041CB69811E4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:539)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:143)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1665)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:864)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:900)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:914)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:873)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:775)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:809)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:820)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=22328, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=22328, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:38108/oav: Could not find collection : awholynewstresscollection_collection2_1
	at __randomizedtesting.SeedInfo.seed([79703BC618647C1C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1099)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:895)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=6584, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6584, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:54364: Could not find collection : awholynewstresscollection_collection4_0
	at __randomizedtesting.SeedInfo.seed([79703BC618647C1C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1099)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:895)


FAILED:  org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
Captured an uncaught exception in thread: Thread[id=68378, name=coreZkRegister-6128-thread-2, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=68378, name=coreZkRegister-6128-thread-2, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
Caused by: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([79703BC618647C1C]:0)
	at org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
	at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:926)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:881)
	at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10261 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/init-core-data-001
   [junit4]   2> 1078488 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1078488 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1078603 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1078624 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1078636 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1078648 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_36226_hdfs____u0pisd/webapp
   [junit4]   2> 1078763 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1079016 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36226
   [junit4]   2> 1079283 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1079284 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1079301 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40556_datanode____.45e0e7/webapp
   [junit4]   2> 1079402 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1079589 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40556
   [junit4]   2> 1079634 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1079635 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1079658 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_39289_datanode____.3v4emk/webapp
   [junit4]   2> 1079798 INFO  (IPC Server handler 3 on 53937) [    ] BlockStateChange BLOCK* processReport: from storage DS-7ba55d40-ca92-4118-b425-07ab32550d0c node DatanodeRegistration(127.0.0.1, datanodeUuid=2eb1b2b8-ef3f-445e-8693-e3e6b9e02fa9, infoPort=40556, ipcPort=36362, storageInfo=lv=-56;cid=testClusterID;nsid=31085310;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1079798 INFO  (IPC Server handler 3 on 53937) [    ] BlockStateChange BLOCK* processReport: from storage DS-037684ac-428a-4159-8b1c-6c2440624643 node DatanodeRegistration(127.0.0.1, datanodeUuid=2eb1b2b8-ef3f-445e-8693-e3e6b9e02fa9, infoPort=40556, ipcPort=36362, storageInfo=lv=-56;cid=testClusterID;nsid=31085310;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1079808 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1080145 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[79703BC618647C1C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39289
   [junit4]   2> 1080422 INFO  (IPC Server handler 5 on 53937) [    ] BlockStateChange BLOCK* processReport: from storage DS-f01edb34-4ea8-4352-a408-decd4b2a44c6 node DatanodeRegistration(127.0.0.1, datanodeUuid=fe2fa4d9-28e9-4359-a499-e86d52ccf5e0, infoPort=39289, ipcPort=32938, storageInfo=lv=-56;cid=testClusterID;nsid=31085310;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1080422 INFO  (IPC Server handler 5 on 53937) [    ] BlockStateChange BLOCK* processReport: from storage DS-05be9c37-aa07-49e8-b1f7-ca758fd1a5fe node DatanodeRegistration(127.0.0.1, datanodeUuid=fe2fa4d9-28e9-4359-a499-e86d52ccf5e0, infoPort=39289, ipcPort=32938, storageInfo=lv=-56;cid=testClusterID;nsid=31085310;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1080540 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1080540 INFO  (Thread-11828) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1080541 INFO  (Thread-11828) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1080641 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ZkTestServer start zk server on port:44893
   [junit4]   2> 1080641 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1080641 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1080644 INFO  (zkCallback-772-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4877cdf0 name:ZooKeeperConnection Watcher:127.0.0.1:44893 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1080644 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1080644 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1080644 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1080647 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1080647 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1080648 INFO  (zkCallback-773-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@54485bd1 name:ZooKeeperConnection Watcher:127.0.0.1:44893/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1080648 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1080649 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1080649 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1080652 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1080654 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1080655 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1080657 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1080657 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1080659 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1080659 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1080660 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1080661 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1080662 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1080662 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1080663 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1080664 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1080665 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1080665 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1080667 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1080667 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1080668 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1080668 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1080670 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1080670 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1080671 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1080671 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1080673 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1080673 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1080987 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores/collection1
   [junit4]   2> 1080988 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1080990 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21354ce0{/,null,AVAILABLE}
   [junit4]   2> 1080990 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@371e207{HTTP/1.1}{127.0.0.1:49192}
   [junit4]   2> 1080990 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.Server Started @1084231ms
   [junit4]   2> 1080990 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:53937/hdfs__localhost_53937__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001_tempDir-002_control_data, hostContext=/, hostPort=49192, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores}
   [junit4]   2> 1080991 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1080991 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/'
   [junit4]   2> 1080991 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1080991 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1081005 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1081006 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081007 INFO  (zkCallback-774-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7e02154a name:ZooKeeperConnection Watcher:127.0.0.1:44893/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081007 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1081007 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1081008 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1081008 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/solr.xml
   [junit4]   2> 1081013 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores
   [junit4]   2> 1081013 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoreContainer New CoreContainer 2110446700
   [junit4]   2> 1081013 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/]
   [junit4]   2> 1081013 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/lib
   [junit4]   2> 1081013 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/lib).
   [junit4]   2> 1081020 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1081021 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1081021 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1081021 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1081022 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44893/solr
   [junit4]   2> 1081022 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1081022 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1081022 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081023 INFO  (zkCallback-776-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1283bb0b name:ZooKeeperConnection Watcher:127.0.0.1:44893 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081024 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1081024 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1081026 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081027 INFO  (zkCallback-777-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4cabb976 name:ZooKeeperConnection Watcher:127.0.0.1:44893/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081027 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1081028 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1081031 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1081034 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1081035 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1081037 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1081039 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1081040 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1081041 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1081043 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1081043 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49192_
   [junit4]   2> 1081044 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:49192_
   [junit4]   2> 1081046 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1081047 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1081048 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1081048 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94682573230047236-127.0.0.1:49192_-n_0000000000
   [junit4]   2> 1081049 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49192_
   [junit4]   2> 1081049 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1081050 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer Overseer (id=94682573230047236-127.0.0.1:49192_-n_0000000000) starting
   [junit4]   2> 1081052 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1081059 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1081059 INFO  (OverseerCollectionConfigSetProcessor-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1081059 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1081060 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1081070 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1081096 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1081096 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1081097 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores
   [junit4]   2> 1081098 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1081098 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores/collection1/
   [junit4]   2> 1081098 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1081100 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1081100 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1081100 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 1081100 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:49192_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1081101 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49192",
   [junit4]   2>   "node_name":"127.0.0.1:49192_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1081102 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1081102 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49192",
   [junit4]   2>   "node_name":"127.0.0.1:49192_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1081102 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1081103 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1081104 INFO  (zkCallback-777-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1081104 INFO  (zkCallback-777-thread-2-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1082102 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1082102 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1082103 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1082103 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 1082104 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1082104 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores/collection1/'
   [junit4]   2> 1082104 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1082104 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1082118 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1082122 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1082126 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1082133 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1082134 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1082138 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1082221 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1082222 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1082222 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1082225 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1082239 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 1082239 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1082239 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 1082240 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53937/solr_hdfs_home
   [junit4]   2> 1082240 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1082240 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1082240 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1adee74e
   [junit4]   2> 1082240 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53937/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1082251 WARN  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 1087257 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:53937/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1087258 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:53937/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 1087267 WARN  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:53937/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1087267 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53937/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1087275 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1087275 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1087280 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1087280 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:53937/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1087281 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=543362292, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1087308 INFO  (IPC Server handler 7 on 53937) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52933 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-f01edb34-4ea8-4352-a408-decd4b2a44c6:NORMAL:127.0.0.1:56089|RBW], ReplicaUnderConstruction[[DISK]DS-7ba55d40-ca92-4118-b425-07ab32550d0c:NORMAL:127.0.0.1:52933|RBW]]} size 0
   [junit4]   2> 1087310 INFO  (IPC Server handler 8 on 53937) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:56089 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-7ba55d40-ca92-4118-b425-07ab32550d0c:NORMAL:127.0.0.1:52933|RBW], ReplicaUnderConstruction[[DISK]DS-05be9c37-aa07-49e8-b1f7-ca758fd1a5fe:NORMAL:127.0.0.1:56089|FINALIZED]]} size 0
   [junit4]   2> 1087313 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b86d5e03 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@58afe95e); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1087313 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1087315 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:53937/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 1087318 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1087319 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1087319 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1087319 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1087319 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1087319 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1087320 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1087320 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1087320 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1087322 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1087323 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1087324 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1087325 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1087329 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1087329 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1087330 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1087330 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 1087341 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 1087341 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 1087342 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1087350 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b86d5e03 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@58afe95e); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1087350 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1087350 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@2ef2ffcd[collection1] main
   [junit4]   2> 1087350 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 1087351 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1087351 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1087351 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1087351 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1087351 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1087352 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1087352 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1087352 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1087352 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1087353 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1087353 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 1087353 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1087353 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1514921178822606848
   [junit4]   2> 1087353 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with highest version 1514921178822606848
   [junit4]   2> 1087353 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1087354 INFO  (coreLoadExecutor-2476-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1087354 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:49192 collection:control_collection shard:shard1
   [junit4]   2> 1087354 INFO  (searcherExecutor-2477-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2ef2ffcd[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1087355 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1087358 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94682573230047236-core_node1-n_0000000000
   [junit4]   2> 1087359 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 1087361 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1087361 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1087361 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1087361 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:49192/collection1/
   [junit4]   2> 1087362 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1087362 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:49192/collection1/ has no replicas
   [junit4]   2> 1087362 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:49192/collection1/ shard1
   [junit4]   2> 1087362 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1087362 INFO  (zkCallback-777-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087363 INFO  (zkCallback-777-thread-2-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087366 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1087368 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:49192",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1087471 INFO  (zkCallback-777-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087471 INFO  (zkCallback-777-thread-2-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087519 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:49192/collection1/ and leader is http://127.0.0.1:49192/collection1/
   [junit4]   2> 1087520 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49192
   [junit4]   2> 1087520 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1087520 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1087520 INFO  (coreZkRegister-2470-thread-1-processing-n:127.0.0.1:49192_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49192_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1087522 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49192",
   [junit4]   2>   "node_name":"127.0.0.1:49192_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1087522 INFO  (OverseerStateUpdate-94682573230047236-127.0.0.1:49192_-n_0000000000) [n:127.0.0.1:49192_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49192",
   [junit4]   2>   "node_name":"127.0.0.1:49192_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1087604 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1087605 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087606 INFO  (zkCallback-779-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@545a468d name:ZooKeeperConnection Watcher:127.0.0.1:44893/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087606 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1087606 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1087606 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1087609 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1087625 INFO  (zkCallback-777-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087625 INFO  (zkCallback-777-thread-2-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087625 INFO  (zkCallback-779-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1087930 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/cores/collection1
   [junit4]   2> 1087931 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001
   [junit4]   2> 1087932 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1087933 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@351b5edd{/,null,AVAILABLE}
   [junit4]   2> 1087934 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@9096c93{HTTP/1.1}{127.0.0.1:50337}
   [junit4]   2> 1087934 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.e.j.s.Server Started @1091175ms
   [junit4]   2> 1087934 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:53937/hdfs__localhost_53937__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=50337, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/cores}
   [junit4]   2> 1087934 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1087934 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/'
   [junit4]   2> 1087934 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1087935 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1087949 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1087950 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087951 INFO  (zkCallback-780-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@63b12ad9 name:ZooKeeperConnection Watcher:127.0.0.1:44893/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087951 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1087951 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1087952 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1087952 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/solr.xml
   [junit4]   2> 1087957 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/cores
   [junit4]   2> 1087957 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoreContainer New CoreContainer 936397902
   [junit4]   2> 1087957 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/]
   [junit4]   2> 1087957 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/lib
   [junit4]   2> 1087957 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/lib).
   [junit4]   2> 1087965 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1087966 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1087966 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1087966 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1087967 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44893/solr
   [junit4]   2> 1087967 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1087967 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1087967 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087968 INFO  (zkCallback-782-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1284ea70 name:ZooKeeperConnection Watcher:127.0.0.1:44893 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087969 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1087970 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1087971 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087972 INFO  (zkCallback-783-thread-1-processing-n:127.0.0.1:50337_) [n:127.0.0.1:50337_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@23c60952 name:ZooKeeperConnection Watcher:127.0.0.1:44893/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087972 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1087977 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1088982 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50337_
   [junit4]   2> 1088982 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:50337_
   [junit4]   2> 1088983 INFO  (zkCallback-777-thread-2-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1088983 INFO  (zkCallback-777-thread-1-processing-n:127.0.0.1:49192_) [n:127.0.0.1:49192_    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1088984 INFO  (zkCallback-779-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1088983 INFO  (zkCallback-783-thread-1-processing-n:127.0.0.1:50337_) [n:127.0.0.1:50337_    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1088985 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1088990 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94682573230047240-127.0.0.1:50337_-n_0000000001
   [junit4]   2> 1088991 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94682573230047236-127.0.0.1:49192_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1089007 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1089007 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1089008 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/cores
   [junit4]   2> 1089009 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[79703BC618647C1C]) [n:127.0.0.1:50337_    ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_79703BC618647C1C-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard

[...truncated too long message...]

:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:197)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:53)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:497)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1665)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 53 more
   [junit4]   2> 3731338 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[79703BC618647C1C]-worker) [n:127.0.0.1:38108_oav c:addReplicaColl s:shard1 r:core_node4 x:addReplicaColl_shard1_replica1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Oct 13, 2015 9:47:44 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ro_RO, timezone=Etc/Greenwich
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_45 (64-bit)/cpus=4,threads=2,free=274685592,total=522715136
   [junit4]   2> NOTE: All tests run in this JVM: [TestBinaryResponseWriter, DistribDocExpirationUpdateProcessorTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestDownShardTolerantSearch, TestSolrQueryParserResource, AtomicUpdatesTest, TestRebalanceLeaders, DocExpirationUpdateProcessorFactoryTest, InfoHandlerTest, SuggestComponentTest, TermsComponentTest, SimplePostToolTest, TestFastOutputStream, TestHashPartitioner, TestFileDictionaryLookup, FastVectorHighlighterTest, ConnectionManagerTest, CachingDirectoryFactoryTest, FullHLLTest, TestJsonRequest, DocumentBuilderTest, TestCodecSupport, TestDistribIDF, TestReloadAndDeleteDocs, TestTrie, SoftAutoCommitTest, TestJettySolrRunner, TestUtils, RuleEngineTest, FacetPivotSmallTest, VMParamsZkACLAndCredentialsProvidersTest, TestStressVersions, MBeansHandlerTest, DirectSolrSpellCheckerTest, TestSolrDeletionPolicy2, CoreAdminCreateDiscoverTest, TestSurroundQueryParser, MultiTermTest, DistribCursorPagingTest, NotRequiredUniqueKeyTest, GraphQueryTest, TestNonDefinedSimilarityFactory, TestReloadDeadlock, RollingRestartTest, SaslZkACLProviderTest, ConvertedLegacyTest, DistributedQueueTest, BlockCacheTest, ChaosMonkeyNothingIsSafeTest, RequestLoggingTest, SolrIndexSplitterTest, DOMUtilTest, TestFiltering, TestRestoreCore, SortByFunctionTest, SpatialRPTFieldTypeTest, TestUpdate, TestRuleBasedAuthorizationPlugin, SpellCheckComponentTest, HdfsLockFactoryTest, SolrPluginUtilsTest, TermVectorComponentTest, SolrInfoMBeanTest, TestSortByMinMaxFunction, TestNRTOpen, BitVectorTest, TestMiniSolrCloudCluster, CircularListTest, TestManagedSchema, TestComplexPhraseQParserPlugin, TestLeaderElectionZkExpiry, TestCopyFieldCollectionResource, TestConfigOverlay, ZkControllerTest, CollectionsAPIDistributedZkTest, HdfsRecoverLeaseTest, TestJmxMonitoredMap, TestMacros, DistributedFacetPivotWhiteBoxTest, SpellCheckCollatorTest, TestManagedSchemaFieldTypeResource, MoreLikeThisHandlerTest, TestCryptoKeys, TestConfigSets, SolrCloudExampleTest, OpenCloseCoreStressTest, TestTolerantSearch, DistributedExpandComponentTest, UniqFieldsUpdateProcessorFactoryTest, TestSchemaManager, SignatureUpdateProcessorFactoryTest, CoreAdminHandlerTest, TestStressLucene, TestCSVLoader, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, TestRemoteStreaming, DocValuesMultiTest, DistanceFunctionTest, XsltUpdateRequestHandlerTest, LukeRequestHandlerTest, TestWriterPerf, TestOmitPositions, RequestHandlersTest, LoggingHandlerTest, IndexSchemaTest, TestCSVResponseWriter, TestPartialUpdateDeduplication, TestComponentsName, SOLR749Test, TestQuerySenderListener, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, TestRTGBase, DistributedIntervalFacetingTest, AssignTest, AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, CdcrReplicationDistributedZkTest, RemoteQueryErrorTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, TestCloudInspectUtil, TestCloudPivotFacet, TestMiniSolrCloudClusterSSL, TestRandomRequestDistribution, TestSolrCloudWithKerberosAlt, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [461/552] on J1 in 290.94s, 1 test, 1 error <<< FAILURES!

[...truncated 297 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987: There were test failures: 552 suites (8 ignored), 2170 tests, 3 errors, 1 failure, 30 ignored (17 assumptions) [seed: 79703BC618647C1C]

Total time: 176 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message