lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-5.5 - Build # 8 - Still Failing
Date Sun, 21 Feb 2016 04:19:17 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.5/8/

3 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:44582/o_lc/awholynewcollection_3: non ok status: 500, message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44582/o_lc/awholynewcollection_3: non ok status: 500, message:Server Error
	at __randomizedtesting.SeedInfo.seed([E4D8A34537722260:6C8C9C9F998E4F98]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:510)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForNon403or404or503(AbstractFullDistribZkTestBase.java:1753)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:737)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:160)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:964)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:939)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest:     1) Thread[id=18481, name=zkCallback-1292-thread-2, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=15517, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-EventThread, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    3) Thread[id=15516, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-SendThread(127.0.0.1:52710), state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)    4) Thread[id=15518, name=zkCallback-1292-thread-1, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest: 
   1) Thread[id=18481, name=zkCallback-1292-thread-2, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=15517, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-EventThread, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   3) Thread[id=15516, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-SendThread(127.0.0.1:52710), state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
   4) Thread[id=15518, name=zkCallback-1292-thread-1, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E4D8A34537722260]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=18481, name=zkCallback-1292-thread-2, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=15516, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-SendThread(127.0.0.1:52710), state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    3) Thread[id=15518, name=zkCallback-1292-thread-1, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=18481, name=zkCallback-1292-thread-2, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=15516, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-SendThread(127.0.0.1:52710), state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   3) Thread[id=15518, name=zkCallback-1292-thread-1, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E4D8A34537722260]:0)




Build Log:
[...truncated 11575 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/init-core-data-001
   [junit4]   2> 1567599 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1567599 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /o_lc/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1567664 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1567676 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1567679 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1567694 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_38063_hdfs____sc8suq/webapp
   [junit4]   2> 1567926 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1568280 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38063
   [junit4]   2> 1568412 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1568416 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1568442 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_41855_datanode____.3z3qla/webapp
   [junit4]   2> 1568746 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1569362 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41855
   [junit4]   2> 1569487 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1569488 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1569512 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_52250_datanode____qoysmv/webapp
   [junit4]   2> 1569625 INFO  (IPC Server handler 9 on 36567) [    ] BlockStateChange BLOCK* processReport: from storage DS-16d5ad67-d099-4066-a092-343b821eabc7 node DatanodeRegistration(127.0.0.1, datanodeUuid=e76c94c9-0213-4ad0-9fc2-20c589a680ea, infoPort=41855, ipcPort=44520, storageInfo=lv=-56;cid=testClusterID;nsid=1624647502;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1569625 INFO  (IPC Server handler 9 on 36567) [    ] BlockStateChange BLOCK* processReport: from storage DS-19e93e90-0d0c-4266-9730-1b9322bd97cf node DatanodeRegistration(127.0.0.1, datanodeUuid=e76c94c9-0213-4ad0-9fc2-20c589a680ea, infoPort=41855, ipcPort=44520, storageInfo=lv=-56;cid=testClusterID;nsid=1624647502;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1569851 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1570442 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4D8A34537722260]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52250
   [junit4]   2> 1570701 INFO  (IPC Server handler 4 on 36567) [    ] BlockStateChange BLOCK* processReport: from storage DS-a42d2179-9d10-4d3f-84f3-9328653ca6bd node DatanodeRegistration(127.0.0.1, datanodeUuid=934661ee-4e6a-4da3-9b43-5aa5b66ad246, infoPort=52250, ipcPort=43161, storageInfo=lv=-56;cid=testClusterID;nsid=1624647502;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 1570702 INFO  (IPC Server handler 4 on 36567) [    ] BlockStateChange BLOCK* processReport: from storage DS-09880459-00bf-4539-a63a-76b926104a59 node DatanodeRegistration(127.0.0.1, datanodeUuid=934661ee-4e6a-4da3-9b43-5aa5b66ad246, infoPort=52250, ipcPort=43161, storageInfo=lv=-56;cid=testClusterID;nsid=1624647502;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1570866 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1570867 INFO  (Thread-9527) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1570867 INFO  (Thread-9527) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1570967 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.ZkTestServer start zk server on port:52710
   [junit4]   2> 1570967 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1570969 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1570974 INFO  (zkCallback-1255-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7eeebd65 name:ZooKeeperConnection Watcher:127.0.0.1:52710 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1570974 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1570975 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1570975 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1570979 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1570980 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1570983 INFO  (zkCallback-1256-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5b04b252 name:ZooKeeperConnection Watcher:127.0.0.1:52710/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1570983 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1570983 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1570983 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1570985 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1570987 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1570988 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1570991 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1570991 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1570993 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1570994 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1570995 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1570996 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1571000 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1571001 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1571010 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1571011 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1571013 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1571013 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1571017 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1571017 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1571020 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1571020 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1571022 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1571022 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1571028 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1571028 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1571032 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1571032 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1571162 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores/collection1
   [junit4]   2> 1571163 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1571168 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9052336{/o_lc,null,AVAILABLE}
   [junit4]   2> 1571169 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.ServerConnector Started ServerConnector@304ab2da{HTTP/1.1}{127.0.0.1:37590}
   [junit4]   2> 1571169 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.Server Started @1573961ms
   [junit4]   2> 1571169 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=37590, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores, hostContext=/o_lc, solr.data.dir=hdfs://localhost:36567/hdfs__localhost_36567__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.5_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001_tempDir-002_control_data}
   [junit4]   2> 1571169 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@41692a49
   [junit4]   2> 1571170 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001'
   [junit4]   2> 1571170 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1571170 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1571170 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1571171 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571172 INFO  (zkCallback-1257-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@74028487 name:ZooKeeperConnection Watcher:127.0.0.1:52710/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571173 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1571173 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1571175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1571175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/solr.xml
   [junit4]   2> 1571191 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores
   [junit4]   2> 1571191 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.CoreContainer New CoreContainer 1468762695
   [junit4]   2> 1571191 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001]
   [junit4]   2> 1571191 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/lib
   [junit4]   2> 1571191 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1571192 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1571193 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1571193 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1571193 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52710/solr
   [junit4]   2> 1571193 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1571193 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1571194 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571196 INFO  (zkCallback-1260-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@40a3005c name:ZooKeeperConnection Watcher:127.0.0.1:52710 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571197 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1571197 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1571199 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571202 INFO  (zkCallback-1261-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1a4115cc name:ZooKeeperConnection Watcher:127.0.0.1:52710/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571202 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1571204 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1571214 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1571219 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1571221 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1571225 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1571227 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1571229 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1571234 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1571235 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1571238 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37590_o_lc
   [junit4]   2> 1571240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:37590_o_lc
   [junit4]   2> 1571243 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1571245 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1571247 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1571248 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95421944129781764-127.0.0.1:37590_o_lc-n_0000000000
   [junit4]   2> 1571249 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37590_o_lc
   [junit4]   2> 1571249 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1571251 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer Overseer (id=95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) starting
   [junit4]   2> 1571255 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1571266 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1571269 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1571269 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1571273 INFO  (OverseerCollectionConfigSetProcessor-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1571282 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1571338 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1571339 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1571342 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores
   [junit4]   2> 1571347 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {collection=control_collection, config=solrconfig.xml, loadOnStartup=true, shard=, coreNodeName=, dataDir=data/, transient=false, configSetProperties=configsetprops.json, name=collection1, schema=schema.xml}
   [junit4]   2> 1571347 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores/collection1
   [junit4]   2> 1571350 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1571357 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1571357 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1571362 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1
   [junit4]   2> 1571363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [n:127.0.0.1:37590_o_lc    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1571363 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:37590_o_lc",
   [junit4]   2>   "base_url":"http://127.0.0.1:37590/o_lc",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1571362 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1571363 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "node_name":"127.0.0.1:37590_o_lc",
   [junit4]   2>   "base_url":"http://127.0.0.1:37590/o_lc",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1571364 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1571364 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1571366 INFO  (zkCallback-1261-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1571369 INFO  (zkCallback-1261-thread-2-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1572364 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1572364 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1572364 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1572364 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1572365 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1572365 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores/collection1'
   [junit4]   2> 1572365 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1572365 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1572367 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1572374 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1572415 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1572452 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1572460 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1572828 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1572839 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1572841 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1572871 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1572875 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1572886 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1572887 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1572890 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1
   [junit4]   2> 1572891 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1572891 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 1572891 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:36567/solr_hdfs_home
   [junit4]   2> 1572891 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1572891 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/control-001/cores/collection1], dataDir=[hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1572892 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@773300dd
   [junit4]   2> 1572893 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1572931 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1572933 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 1572954 WARN  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:36567/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1572954 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1572972 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1572972 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=0.529296875, floorSegmentMB=1.4658203125, forceMergeDeletesPctAllowed=12.25725889016759, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1573016 INFO  (IPC Server handler 8 on 36567) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39647 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-16d5ad67-d099-4066-a092-343b821eabc7:NORMAL:127.0.0.1:33914|RBW], ReplicaUnderConstruction[[DISK]DS-09880459-00bf-4539-a63a-76b926104a59:NORMAL:127.0.0.1:39647|RBW]]} size 0
   [junit4]   2> 1573019 INFO  (IPC Server handler 4 on 36567) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33914 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-16d5ad67-d099-4066-a092-343b821eabc7:NORMAL:127.0.0.1:33914|RBW], ReplicaUnderConstruction[[DISK]DS-09880459-00bf-4539-a63a-76b926104a59:NORMAL:127.0.0.1:39647|RBW]]} size 0
   [junit4]   2> 1573025 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17426eaa; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1573025 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1573031 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 1573041 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1573042 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1573042 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1573042 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1573043 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1573043 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1573043 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1573046 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1573046 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1573046 WARN  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1573049 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1573051 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1573054 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1573057 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1573063 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/logging,/admin/plugins,/admin/luke,/admin/mbeans,/update/json/docs,/update,/schema,/admin/system,/admin/properties,/replication,standard,/get,/update/json,/admin/file,/admin/segments,/dump,/update/csv,/admin/threads,/admin/ping,/config
   [junit4]   2> 1573067 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1573067 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1573067 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 1573086 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1573087 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1573088 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=1.5849609375, floorSegmentMB=1.203125, forceMergeDeletesPctAllowed=7.968270734387134, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1573100 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:36567/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17426eaa; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1573100 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1573100 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@459cd99a[collection1] main
   [junit4]   2> 1573100 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1573101 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1573101 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1573106 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1573106 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1573106 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1573107 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1573107 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1573107 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1573107 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1573108 INFO  (searcherExecutor-2940-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@459cd99a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1573109 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1573109 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 1573110 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1573110 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1526751108425318400
   [junit4]   2> 1573110 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with highest version 1526751108425318400
   [junit4]   2> 1573110 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1573111 INFO  (coreLoadExecutor-2939-thread-1-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1573114 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:37590/o_lc collection:control_collection shard:shard1
   [junit4]   2> 1573116 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1573123 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95421944129781764-core_node1-n_0000000000
   [junit4]   2> 1573123 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1573126 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1573126 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1573126 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1573126 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37590/o_lc/collection1/
   [junit4]   2> 1573126 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1573126 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:37590/o_lc/collection1/ has no replicas
   [junit4]   2> 1573127 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1573127 INFO  (zkCallback-1261-thread-3-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573127 INFO  (zkCallback-1261-thread-2-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573133 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1573137 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37590/o_lc/collection1/ shard1
   [junit4]   2> 1573137 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:37590/o_lc",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1573240 INFO  (zkCallback-1261-thread-3-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573240 INFO  (zkCallback-1261-thread-2-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573288 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:37590/o_lc/collection1/ and leader is http://127.0.0.1:37590/o_lc/collection1/
   [junit4]   2> 1573288 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37590/o_lc
   [junit4]   2> 1573288 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1573288 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1573288 INFO  (coreZkRegister-2933-thread-1-processing-n:127.0.0.1:37590_o_lc s:shard1 r:core_node1 x:collection1 c:control_collection) [n:127.0.0.1:37590_o_lc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1573290 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:37590_o_lc",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "base_url":"http://127.0.0.1:37590/o_lc",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1573291 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "node_name":"127.0.0.1:37590_o_lc",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "base_url":"http://127.0.0.1:37590/o_lc",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1573365 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1573365 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1573373 INFO  (zkCallback-1263-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@77ac13cb name:ZooKeeperConnection Watcher:127.0.0.1:52710/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1573373 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1573374 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1573374 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1573378 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1573378 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1573378 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1573379 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1573380 INFO  (zkCallback-1264-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@38275ca6 name:ZooKeeperConnection Watcher:127.0.0.1:52710/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1573381 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1573381 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1573383 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 1573384 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1573387 INFO  (zkCallback-1261-thread-2-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573387 INFO  (zkCallback-1261-thread-3-processing-n:127.0.0.1:37590_o_lc) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573390 INFO  (zkCallback-1263-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1573491 INFO  (OverseerStateUpdate-95421944129781764-127.0.0.1:37590_o_lc-n_0000000000) [n:127.0.0.1:37590_o_lc    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1573549 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/shard-1-001/cores/collection1
   [junit4]   2> 1573550 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/shard-1-001
   [junit4]   2> 1573551 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1573555 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1c333d14{/o_lc,null,AVAILABLE}
   [junit4]   2> 1573556 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.ServerConnector Started ServerConnector@67eefa82{HTTP/1.1}{127.0.0.1:44582}
   [junit4]   2> 1573556 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.e.j.s.Server Started @1576348ms
   [junit4]   2> 1573556 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, hostPort=44582, hostContext=/o_lc, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/shard-1-001/cores, solr.data.dir=hdfs://localhost:36567/hdfs__localhost_36567__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.5_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001_tempDir-002_jetty1}
   [junit4]   2> 1573556 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@41692a49
   [junit4]   2> 1573557 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4D8A34537722260-001/shard-1-001'
   [junit4]   2> 1573557 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1573557 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1573557 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1573568 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1573569 INFO  (zkCallback-1265-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.c

[...truncated too long message...]

Stack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {}, docValues:{}, sim=DefaultSimilarity, locale=es-SV, timezone=Africa/Bissau
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=5,free=201027592,total=475004928
   [junit4]   2> NOTE: All tests run in this JVM: [AnalysisErrorHandlingTest, ConvertedLegacyTest, ZkControllerTest, CheckHdfsIndexTest, TestDefaultStatsCache, TestBinaryResponseWriter, WordBreakSolrSpellCheckerTest, TestNonDefinedSimilarityFactory, StressHdfsTest, TestRuleBasedAuthorizationPlugin, RollingRestartTest, TimeZoneUtilsTest, TestCollationFieldDocValues, TestCustomSort, TestSortByMinMaxFunction, SortByFunctionTest, BlockCacheTest, CollectionTooManyReplicasTest, AliasIntegrationTest, TestPartialUpdateDeduplication, HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsCollectionsAPIDistributedZkTest -Dtests.seed=E4D8A34537722260 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es-SV -Dtests.timezone=Africa/Bissau -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | HdfsCollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest: 
   [junit4]    >    1) Thread[id=18481, name=zkCallback-1292-thread-2, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=15517, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-EventThread, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    3) Thread[id=15516, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-SendThread(127.0.0.1:52710), state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
   [junit4]    >    4) Thread[id=15518, name=zkCallback-1292-thread-1, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4D8A34537722260]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=18481, name=zkCallback-1292-thread-2, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=15516, name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4D8A34537722260]-SendThread(127.0.0.1:52710), state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    3) Thread[id=15518, name=zkCallback-1292-thread-1, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4D8A34537722260]:0)
   [junit4] Completed [160/566 (1!)] on J1 in 111.49s, 1 test, 3 errors <<< FAILURES!

[...truncated 1299 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/build.xml:757: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/build.xml:694: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/solr/common-build.xml:534: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/lucene/common-build.xml:1477: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.5/lucene/common-build.xml:1033: There were test failures: 566 suites (7 ignored), 2275 tests, 2 suite-level errors, 1 error, 1321 ignored (12 assumptions) [seed: E4D8A34537722260]

Total time: 259 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message