lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 733 - Still Failing
Date Wed, 08 Jul 2015 07:09:33 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/733/

3 tests failed.
REGRESSION:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 30 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 30 seconds and no jetties were stopped - this is worth investigating!
	at __randomizedtesting.SeedInfo.seed([9CD33C16F091D81F:148703CC5E6DB5E7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:537)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:143)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:41779/db: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41779/db: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:301)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:418)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:168)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=91932, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=91932, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:43666: Could not find collection : awholynewstresscollection_collection2_1
	at __randomizedtesting.SeedInfo.seed([9CD33C16F091D81F]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 10393 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/init-core-data-001
   [junit4]   2> 942906 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 942906 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /i/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 942946 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 942958 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 942968 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 942982 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_49849_hdfs____.ttkb3y/webapp
   [junit4]   2> 943113 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 943683 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49849
   [junit4]   2> 943861 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 943862 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 943874 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45248_datanode____jjzvy4/webapp
   [junit4]   2> 944026 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 944947 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45248
   [junit4]   2> 945002 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 945003 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 945033 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_54138_datanode____xt4c72/webapp
   [junit4]   2> 945106 INFO  (IPC Server handler 2 on 49516) [    ] BlockStateChange BLOCK* processReport: from storage DS-47bf18b7-d3f3-4d8b-9f96-e54b5e758769 node DatanodeRegistration(127.0.0.1, datanodeUuid=75c00495-f557-4e5d-8fa0-41c4233d82f0, infoPort=45248, ipcPort=47254, storageInfo=lv=-56;cid=testClusterID;nsid=40507542;c=0), blocks: 0, hasStaleStorages: true, processing time: 3 msecs
   [junit4]   2> 945107 INFO  (IPC Server handler 2 on 49516) [    ] BlockStateChange BLOCK* processReport: from storage DS-f34ce943-4c02-41b7-a662-46f402f762e0 node DatanodeRegistration(127.0.0.1, datanodeUuid=75c00495-f557-4e5d-8fa0-41c4233d82f0, infoPort=45248, ipcPort=47254, storageInfo=lv=-56;cid=testClusterID;nsid=40507542;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 945181 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 945785 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[9CD33C16F091D81F]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54138
   [junit4]   2> 945863 INFO  (IPC Server handler 3 on 49516) [    ] BlockStateChange BLOCK* processReport: from storage DS-903cd247-75e9-4131-b384-ba5082f93343 node DatanodeRegistration(127.0.0.1, datanodeUuid=05aa3e1b-6fb5-4d75-92a1-e1471e1dd0d1, infoPort=54138, ipcPort=59881, storageInfo=lv=-56;cid=testClusterID;nsid=40507542;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 945864 INFO  (IPC Server handler 3 on 49516) [    ] BlockStateChange BLOCK* processReport: from storage DS-cdcf3d6a-fdee-444a-8a46-6621cdfdf7f6 node DatanodeRegistration(127.0.0.1, datanodeUuid=05aa3e1b-6fb5-4d75-92a1-e1471e1dd0d1, infoPort=54138, ipcPort=59881, storageInfo=lv=-56;cid=testClusterID;nsid=40507542;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 945953 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 945953 INFO  (Thread-3827) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 945954 INFO  (Thread-3827) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 946054 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ZkTestServer start zk server on port:48362
   [junit4]   2> 946054 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 946055 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 946065 INFO  (zkCallback-365-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@322be1f1 name:ZooKeeperConnection Watcher:127.0.0.1:48362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946065 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 946065 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 946065 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 946071 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 946072 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 946075 INFO  (zkCallback-366-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@285bab4 name:ZooKeeperConnection Watcher:127.0.0.1:48362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946076 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 946076 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 946076 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 946079 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 946080 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 946081 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 946083 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 946083 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 946085 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 946086 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 946089 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 946089 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 946091 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 946091 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 946092 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 946093 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 946094 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 946095 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 946096 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 946097 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 946099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 946100 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 946103 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 946103 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 946105 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 946105 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 946107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 946108 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 946448 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores/collection1
   [junit4]   2> 946450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 946451 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5dd1df20{/i,null,AVAILABLE}
   [junit4]   2> 946452 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.ServerConnector Started ServerConnector@65cc78ba{HTTP/1.1}{127.0.0.1:46915}
   [junit4]   2> 946452 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.Server Started @949707ms
   [junit4]   2> 946452 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49516/hdfs__localhost_49516__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001_tempDir-002_control_data, hostContext=/i, hostPort=46915, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores}
   [junit4]   2> 946453 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 946453 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/'
   [junit4]   2> 946467 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/solr.xml
   [junit4]   2> 946472 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores
   [junit4]   2> 946472 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoreContainer New CoreContainer 1040738979
   [junit4]   2> 946472 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/]
   [junit4]   2> 946472 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/lib
   [junit4]   2> 946472 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/lib).
   [junit4]   2> 946479 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 946480 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 946481 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 946481 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 946481 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48362/solr
   [junit4]   2> 946481 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 946481 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 946482 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 946484 INFO  (zkCallback-368-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14813fe0 name:ZooKeeperConnection Watcher:127.0.0.1:48362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946485 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 946485 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 946487 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x14e6c3716180002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 946487 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 946490 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d815694 name:ZooKeeperConnection Watcher:127.0.0.1:48362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 946491 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 946493 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 946495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 946496 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 946498 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 946500 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 946502 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 946503 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 946504 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 946505 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46915_i
   [junit4]   2> 946506 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:46915_i
   [junit4]   2> 946507 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 946508 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 946510 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 946510 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94131626071031811-127.0.0.1:46915_i-n_0000000000
   [junit4]   2> 946511 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46915_i
   [junit4]   2> 946511 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 946512 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer Overseer (id=94131626071031811-127.0.0.1:46915_i-n_0000000000) starting
   [junit4]   2> 946514 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 946519 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 946519 INFO  (OverseerCollectionProcessor-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 946519 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 946520 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 946523 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 946523 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 946524 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores
   [junit4]   2> 946525 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 946526 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores/collection1/
   [junit4]   2> 946526 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 946526 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 946526 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 946527 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 946527 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:46915_i    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 946527 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 946527 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 946528 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46915/i",
   [junit4]   2>   "node_name":"127.0.0.1:46915_i",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 946528 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46915/i",
   [junit4]   2>   "node_name":"127.0.0.1:46915_i",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 946528 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 946529 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 946530 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 946531 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 947527 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 947527 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 947528 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 947528 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 947529 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 947529 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores/collection1/'
   [junit4]   2> 947544 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 947548 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 947566 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 947573 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 947575 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 947579 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 947647 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 947647 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 947648 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 947651 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 947664 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 947664 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 947664 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49516/solr_hdfs_home
   [junit4]   2> 947664 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 947664 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 947665 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@74817fc2
   [junit4]   2> 947665 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 947700 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data for locality metrics.
   [junit4]   2> 947700 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 947702 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 947710 WARN  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:49516/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 947711 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 947718 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 947718 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 947718 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 947731 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 947731 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data/index for locality metrics.
   [junit4]   2> 947731 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 947732 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.00390625, noCFSRatio=0.25892960546340227]
   [junit4]   2> 947754 INFO  (IPC Server handler 8 on 49516) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42080 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-f34ce943-4c02-41b7-a662-46f402f762e0:NORMAL:127.0.0.1:37549|RBW], ReplicaUnderConstruction[[DISK]DS-903cd247-75e9-4131-b384-ba5082f93343:NORMAL:127.0.0.1:42080|FINALIZED]]} size 0
   [junit4]   2> 947756 INFO  (IPC Server handler 9 on 49516) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37549 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-f34ce943-4c02-41b7-a662-46f402f762e0:NORMAL:127.0.0.1:37549|RBW], ReplicaUnderConstruction[[DISK]DS-903cd247-75e9-4131-b384-ba5082f93343:NORMAL:127.0.0.1:42080|FINALIZED]]} size 0
   [junit4]   2> 947761 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b7e9e3b2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15521e66); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 947761 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 947762 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:49516/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 947765 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 947765 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 947765 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 947765 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 947766 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 947766 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 947766 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 947766 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 947766 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 947768 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 947769 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 947770 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 947771 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 947781 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 947781 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 947782 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 947782 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 947792 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 947792 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 947793 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=6.85546875, floorSegmentMB=1.822265625, forceMergeDeletesPctAllowed=29.77765088559873, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3241432425070051
   [junit4]   2> 947801 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b7e9e3b2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15521e66); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 947801 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 947802 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@520b585b[collection1] main
   [junit4]   2> 947803 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 947804 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 947804 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 947805 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 947805 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 947805 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 947805 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 947805 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 947806 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 947806 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 947807 INFO  (searcherExecutor-1894-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@520b585b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 947811 WARN  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Could not retrieve locality information for hdfs://localhost:54281/solr3 due to exception: java.net.ConnectException: Call From lucene1-us-west/10.41.0.5 to localhost:54281 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
   [junit4]   2> 947811 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 947812 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 947812 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 947812 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1506106019083714560
   [junit4]   2> 947812 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1 ms to seed version buckets with highest version 1506106019083714560
   [junit4]   2> 947812 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 947813 INFO  (coreLoadExecutor-1893-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 947813 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:46915/i collection:control_collection shard:shard1
   [junit4]   2> 947814 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 947818 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94131626071031811-core_node1-n_0000000000
   [junit4]   2> 947825 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 947827 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 947828 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 947828 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 947828 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46915/i/collection1/
   [junit4]   2> 947828 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 947828 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 947828 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46915/i/collection1/ has no replicas
   [junit4]   2> 947829 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46915/i/collection1/ shard1
   [junit4]   2> 947829 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 947832 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 947833 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46915/i",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 947935 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 947937 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 947983 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:46915/i/collection1/ and leader is http://127.0.0.1:46915/i/collection1/
   [junit4]   2> 947984 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46915/i
   [junit4]   2> 947984 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 947984 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 947984 INFO  (coreZkRegister-1887-thread-1-processing-n:127.0.0.1:46915_i x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46915_i c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 947985 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 947986 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46915/i",
   [junit4]   2>   "node_name":"127.0.0.1:46915_i",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 947987 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46915/i",
   [junit4]   2>   "node_name":"127.0.0.1:46915_i",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 948029 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 948030 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 948032 INFO  (zkCallback-371-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ae72f98 name:ZooKeeperConnection Watcher:127.0.0.1:48362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 948032 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 948033 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 948033 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 948035 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 948035 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 948036 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 948036 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 948045 INFO  (zkCallback-372-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4a6b394a name:ZooKeeperConnection Watcher:127.0.0.1:48362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 948045 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 948045 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 948047 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 948048 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 948048 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 948049 INFO  (zkCallback-371-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 948049 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 948051 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 948053 INFO  (zkCallback-371-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 948153 INFO  (OverseerStateUpdate-94131626071031811-127.0.0.1:46915_i-n_0000000000) [n:127.0.0.1:46915_i    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 948154 INFO  (zkCallback-371-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 948154 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 948156 INFO  (zkCallback-371-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 948156 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46915_i) [n:127.0.0.1:46915_i    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 948376 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/cores/collection1
   [junit4]   2> 948377 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001
   [junit4]   2> 948378 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 948380 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@46667580{/i,null,AVAILABLE}
   [junit4]   2> 948381 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3e8da3d4{HTTP/1.1}{127.0.0.1:50921}
   [junit4]   2> 948381 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.e.j.s.Server Started @951636ms
   [junit4]   2> 948381 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49516/hdfs__localhost_49516__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/i, hostPort=50921, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/cores}
   [junit4]   2> 948381 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 948382 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/'
   [junit4]   2> 948396 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/solr.xml
   [junit4]   2> 948401 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/cores
   [junit4]   2> 948401 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoreContainer New CoreContainer 119478646
   [junit4]   2> 948401 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/]
   [junit4]   2> 948402 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/lib
   [junit4]   2> 948402 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_9CD33C16F091D81F-001/shard-1-001/lib).
   [junit4]   2> 948409 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 948410 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 948410 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 948411 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 948411 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48362/solr
   [junit4]   2> 948411 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 948411 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 948412 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 948414 INFO  (zkCallback-374-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4364797f name:ZooKeeperConnection Watcher:127.0.0.1:48362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 948414 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 948415 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 948417 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:50921_i    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 948419 INFO  (zkCallback-375-thread-1-processing-n:127.0.0.1:50921_i) [n:127.0.0.1:50921_i    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e79a24a name:ZooKeeperConnection Watcher:127.0.0.1:48362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 948419 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:50921_i    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 948426 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:50921_i    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 949430 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F091D81F]) [n:127.0.0.1:50921_i    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50921_i
   [junit4]   2> 949430 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[9CD33C16F

[...truncated too long message...]

ava:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:197)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:50)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:483)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 53 more
   [junit4]   2> 5570189 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[9CD33C16F091D81F]-worker) [n:127.0.0.1:43666_ c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica3] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Jul 08, 2015 3:07:31 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene53, sim=DefaultSimilarity, locale=th, timezone=US/Alaska
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=4,threads=2,free=142078784,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [TestMiniSolrCloudClusterSSL, TestCopyFieldCollectionResource, TestSolrConfigHandlerConcurrent, TestCursorMarkWithoutUniqueKey, TestReplicationHandlerBackup, TestSolrQueryParser, TestRawResponseWriter, TestCollectionAPI, TestMaxScoreQueryParser, DistributedSuggestComponentTest, DistribJoinFromCollectionTest, TestSolrCloudWithKerberosAlt, DeleteReplicaTest, FullSolrCloudDistribCmdsTest, TestQuerySenderNoQuery, FileBasedSpellCheckerTest, StatsComponentTest, SimpleCollectionCreateDeleteTest, TestPerFieldSimilarity, TestLRUStatsCache, TestHashQParserPlugin, AddBlockUpdateTest, BinaryUpdateRequestHandlerTest, OverseerTest, TestReqParamsAPI, RegexBoostProcessorTest, TestDynamicFieldCollectionResource, UnloadDistributedZkTest, SOLR749Test, XsltUpdateRequestHandlerTest, SimpleFacetsTest, PluginInfoTest, HdfsSyncSliceTest, PolyFieldTest, TestQueryTypes, CollectionReloadTest, HdfsDirectoryTest, TestStressReorder, OpenExchangeRatesOrgProviderTest, StressHdfsTest, RecoveryZkTest, RuleEngineTest, DirectSolrSpellCheckerTest, HdfsLockFactoryTest, TestExtendedDismaxParser, TestZkChroot, TestQueryUtils, TestShortCircuitedRequests, LeaderInitiatedRecoveryOnCommitTest, LeaderElectionTest, HdfsNNFailoverTest, CircularListTest, TestChildDocTransformer, BasicDistributedZk2Test, TestCollationField, TestTrie, TestSmileRequest, DirectUpdateHandlerOptimizeTest, VersionInfoTest, SharedFSAutoReplicaFailoverTest, DistributedVersionInfoTest, TestInitParams, CloudExitableDirectoryReaderTest, DistributedQueryElevationComponentTest, TestBM25SimilarityFactory, TestFieldTypeResource, TestDocSet, TestDefaultSimilarityFactory, CustomCollectionTest, SynonymTokenizerTest, DirectUpdateHandlerTest, TestPartialUpdateDeduplication, TestXIncludeConfig, ReplicationFactorTest, TestSolrQueryParserDefaultOperatorResource, TestSolr4Spatial2, SearchHandlerTest, DOMUtilTest, TestAuthorizationFramework, TestBinaryResponseWriter, ClusterStateUpdateTest, TestRandomFaceting, ZkSolrClientTest, ZkCLITest, TestDistributedSearch, TestSolrDeletionPolicy2, SampleTest, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, EchoParamsTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, DistributedMLTComponentTest, TestRTGBase, CursorPagingTest, DistributedIntervalFacetingTest, AsyncMigrateRouteKeyTest, ConcurrentDeleteAndCreateCollectionTest, ConnectionManagerTest, DeleteShardTest, MigrateRouteKeyTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerStatusTest, RecoveryAfterSoftCommitTest, RemoteQueryErrorTest, SSLMigrationTest, SaslZkACLProviderTest, ShardSplitTest, SharedFSAutoReplicaFailoverUtilsTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudPivotFacet, TestCryptoKeys, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry, TestRandomRequestDistribution, TestReplicaProperties, TestRequestStatusCollectionAPI, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [514/514] on J2 in 1945.34s, 1 test, 1 error <<< FAILURES!

[...truncated 13 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:533: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:513: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1426: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:981: There were test failures: 514 suites, 2022 tests, 2 errors, 1 failure, 20 ignored (10 assumptions)

Total time: 197 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message