lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1032 - Still Failing
Date Sat, 04 Jun 2016 15:09:10 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1032/

20 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:41801

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:41801
	at __randomizedtesting.SeedInfo.seed([AA07A683C22DBAA5:225399596CD1D75D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:617)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoCollectionSpecified(CollectionsAPIDistributedZkTest.java:574)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReqParamsAPI

Error Message:


Stack Trace:
java.lang.AssertionError
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReqParamsAPI

Error Message:
Could not remove the following files (in the order of attempts):    /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001/cores: java.nio.file.DirectoryNotEmptyException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001/cores    /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001: java.nio.file.DirectoryNotEmptyException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001/cores: java.nio.file.DirectoryNotEmptyException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001/cores
   /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001: java.nio.file.DirectoryNotEmptyException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_AA07A683C22DBAA5-001/shard-4-001

	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReqParamsAPI

Error Message:
107 threads leaked from SUITE scope at org.apache.solr.handler.TestReqParamsAPI:     1) Thread[id=16832, name=coreZkRegister-2938-thread-15-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:987)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:913)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=16857, name=coreZkRegister-2971-thread-23-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:987)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:913)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=16799, name=OverseerStateUpdate-96011431611203810-127.0.0.1:58268__t%2Fis-n_0000000006, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:168)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:142)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:196)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=16772, name=coreCloseExecutor-4292-thread-8, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=16849, name=coreZkRegister-2971-thread-15-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:987)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:913)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=16802, name=coreZkRegister-2938-thread-2-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=16801, name=OverseerHdfsCoreFailoverThread-96011431611203810-127.0.0.1:58268__t%2Fis-n_0000000006, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=16845, name=coreZkRegister-2938-thread-28-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=16807, name=coreZkRegister-2971-thread-6-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=16781, name=coreCloseExecutor-4292-thread-17, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=16800, name=OverseerCollectionConfigSetProcessor-96011431611203810-127.0.0.1:58268__t%2Fis-n_0000000006, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1153)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:328)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:167)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=16870, name=zkCallback-1913-thread-34, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=16765, name=coreCloseExecutor-4292-thread-1, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=16827, name=coreZkRegister-2938-thread-10, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=16814, name=zkCallback-1901-thread-34, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=16822, name=coreZkRegister-2938-thread-5-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=16846, name=coreZkRegister-2938-thread-29-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=16798, name=watches-1914-thread-6, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=16793, name=watches-1914-thread-4, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=16810, name=coreZkRegister-2971-thread-9, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=16842, name=coreZkRegister-2938-thread-25-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=16835, name=coreZkRegister-2938-thread-18-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=16811, name=coreZkRegister-2971-thread-10-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=16852, name=coreZkRegister-2971-thread-18-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=16851, name=coreZkRegister-2971-thread-17-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=16804, name=coreZkRegister-2971-thread-3-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=16829, name=coreZkRegister-2938-thread-12, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=16790, name=coreCloseExecutor-4292-thread-26, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=16766, name=coreCloseExecutor-4292-thread-2, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=16820, name=coreZkRegister-2938-thread-3-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=16778, name=coreCloseExecutor-4292-thread-14, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=16773, name=coreCloseExecutor-4292-thread-9, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=16809, name=coreZkRegister-2971-thread-8-processing-n:127.0.0.1:34686__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)         at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:579)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=16828, name=coreZkRegister-2938-thread-11, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=16843, name=coreZkRegister-2938-thread-26-processing-n:127.0.0.1:50398__t%2Fis, state=BLOCKED, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:62)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.cloud.ZkController.register(ZkController.java:807)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=16836, name=coreZkRegister-2938-thread-19-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=16826, name=coreZkRegister-2938-thread-9-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1153)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:971)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:957)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:913)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=16824, name=coreZkRegister-2938-thread-7-processing-n:127.0.0.1:50398__t%2Fis, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1029)         at org.apache.solr.cloud.ZkController.register(ZkController.java:840)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:204)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=16754, name=qtp2100115844-16754, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=16777, name=coreCloseExecutor-4292-thread-13, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=16784, name=coreCloseExecutor-4292-thread-20, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.hadoop.ipc.Client.call(Client.java:1454)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy45.getListing(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy46.getListing(Unknown Source)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)         at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)         at org.apache.solr.store.hdfs.HdfsDirectory.listAll(HdfsDirectory.java:173)         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)         at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.ja

[...truncated too long message...]

pache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)
   [junit4]    >         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2130)
   [junit4]    >         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2072)
   [junit4]    >         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1103)
   [junit4]    >         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1145)
   [junit4]    >         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:130)
   [junit4]    >         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:832)
   [junit4]    >         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:88)
   [junit4]    >         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:376)
   [junit4]    >         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:74)
   [junit4]    >         at org.apache.solr.core.SolrCore.close(SolrCore.java:1171)
   [junit4]    >         at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:131)
   [junit4]    >         at org.apache.solr.core.SolrCores$$Lambda$58/901913193.call(Unknown Source)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$9/685712043.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   93) Thread[id=16827, name=coreZkRegister-2938-thread-10, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   94) Thread[id=16814, name=zkCallback-1901-thread-34, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   95) Thread[id=16822, name=coreZkRegister-2938-thread-5, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   96) Thread[id=16847, name=coreZkRegister-2938-thread-30, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AA07A683C22DBAA5]:0)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11418, name=CacheReplicationMonitor(241816060), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.hadoop.util.ExitUtil$ExitException: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
   [junit4]    > 	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:210)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11739, name=OverseerThreadFactory-2930-thread-1, state=RUNNABLE, group=Overseer collection creation process.]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11542, name=OverseerCollectionConfigSetProcessor-96011424879280135-127.0.0.1:58268__t%2Fis-n_0000000000, state=RUNNABLE, group=Overseer collection creation process.]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4] Completed [538/612 (10!)] on J1 in 111.43s, 0 tests, 1 failure, 6 errors <<< FAILURES!

[...truncated 220 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160604_050504_057.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid12662.hprof ...
   [junit4] Heap dump file created [683547685 bytes in 30.136 secs]
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160604_050504_057.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "IPC Server idle connection scanner for port 53520"
   [junit4] <<< JVM J1: EOF ----

[...truncated 19 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:531: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1427: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2559 tests, 18 suite-level errors, 3 errors, 303 ignored (16 assumptions) [seed: AA07A683C22DBAA5]

Total time: 784 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




Mime
View raw message