lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_172) - Build # 2721 - Unstable!
Date Mon, 10 Sep 2018 18:56:30 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2721/
Java: 64bit/jdk1.8.0_172 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.ltr.TestLTROnSolrCloud

Error Message:
45 threads leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud:     1) Thread[id=45, name=OverseerStateUpdate-72201792901742595-127.0.0.1:42417_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=72, name=Scheduler-1737580576, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=82, name=qtp2024767747-82, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=54, name=ScheduledTrigger-7-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=55, name=ScheduledTrigger-7-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=89, name=SolrRrdBackendFactory-11-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=46, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=48, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    9) Thread[id=31, name=qtp2024767747-31, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)         at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)         at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)         at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)         at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)         at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)         at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)         at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)         at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)         at org.eclipse.jetty.server.Server.handle(Server.java:531)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)         at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=44, name=zkConnectionManagerCallback-12-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=38, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   12) Thread[id=57, name=ScheduledTrigger-7-thread-3, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   13) Thread[id=36, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=25, name=qtp2024767747-25, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=32, name=Scheduler-696875663, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=70, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[B982107109F46690]-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   17) Thread[id=65, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=26, name=qtp2024767747-26, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=50, name=OverseerCollectionConfigSetProcessor-72201792901742595-127.0.0.1:42417_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)         at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=14, name=Thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)   21) Thread[id=59, name=MetricsHistoryHandler-12-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   22) Thread[id=16, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)   23) Thread[id=29, name=qtp2024767747-29, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   24) Thread[id=42, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:39081), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)   25) Thread[id=30, name=qtp2024767747-30, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.lang.Thread.run(Thread.java:748)   26) Thread[id=27, name=qtp2024767747-27, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)         at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)         at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)         at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)         at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)         at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)         at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)         at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)         at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)         at org.eclipse.jetty.server.Server.handle(Server.java:531)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)         at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$30/1333528501.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   27) Thread[id=51, name=OverseerAutoScalingTriggerThread-72201792901742595-127.0.0.1:42417_solr-n_0000000000, state=WAITING, group=Overseer autoscaling triggers]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)         at java.lang.Thread.run(Thread.java:748)   28) Thread[id=47, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:39081), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)   29) Thread[id=58, name=ScheduledTrigger-7-thread-4, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   30) Thread[id=43, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   31) Thread[id=60, name=Thread-11, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)   32) Thread[id=81, name=qtp2024767747-81, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   33) Thread[id=15, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.lang.Thread.run(Thread.java:748)   34) Thread[id=69, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[B982107109F46690]-SendThread(127.0.0.1:39081), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)   35) Thread[id=71, name=zkConnectionManagerCallback-29-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   36) Thread[id=49, name=zkConnectionManagerCallback-19-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   37) Thread[id=18, name=ProcessThread(sid:0 cport:39081):, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)   38) Thread[id=83, name=qtp2024767747-83, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)         at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)         at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)         at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)         at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)         at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)         at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)         at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)         at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)         at org.eclipse.jetty.server.Server.handle(Server.java:531)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)         at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   39) Thread[id=56, name=SolrRrdBackendFactory-11-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   40) Thread[id=79, name=qtp2024767747-79, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)         at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)         at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)         at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)         at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)         at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)         at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)         at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)         at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)         at org.eclipse.jetty.server.Server.handle(Server.java:531)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)         at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   41) Thread[id=84, name=qtp2024767747-84, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   42) Thread[id=37, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   43) Thread[id=73, name=OverseerThreadFactory-9-thread-1-processing-n:127.0.0.1:42417_solr, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)         at org.apache.solr.handler.component.HttpShardHandler.take(HttpShardHandler.java:248)         at org.apache.solr.handler.component.HttpShardHandler.takeCompletedOrError(HttpShardHandler.java:241)         at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processResponses(OverseerCollectionMessageHandler.java:728)         at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:290)         at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:259)         at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:478)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$10/1419318953.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   44) Thread[id=24, name=qtp2024767747-24, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)   45) Thread[id=28, name=qtp2024767747-28-acceptor-0@7ed7f153-ServerConnector@701010f9{SSL,[ssl, http/1.1]}{127.0.0.1:42417}, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 45 threads leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud: 
   1) Thread[id=45, name=OverseerStateUpdate-72201792901742595-127.0.0.1:42417_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=72, name=Scheduler-1737580576, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=82, name=qtp2024767747-82, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=54, name=ScheduledTrigger-7-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=55, name=ScheduledTrigger-7-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=89, name=SolrRrdBackendFactory-11-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=46, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=48, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   9) Thread[id=31, name=qtp2024767747-31, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)
        at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)
        at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
        at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=44, name=zkConnectionManagerCallback-12-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=38, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  12) Thread[id=57, name=ScheduledTrigger-7-thread-3, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=36, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=25, name=qtp2024767747-25, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=32, name=Scheduler-696875663, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  16) Thread[id=70, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[B982107109F46690]-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  17) Thread[id=65, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  18) Thread[id=26, name=qtp2024767747-26, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  19) Thread[id=50, name=OverseerCollectionConfigSetProcessor-72201792901742595-127.0.0.1:42417_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
        at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)
        at java.lang.Thread.run(Thread.java:748)
  20) Thread[id=14, name=Thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)
  21) Thread[id=59, name=MetricsHistoryHandler-12-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  22) Thread[id=16, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
  23) Thread[id=29, name=qtp2024767747-29, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  24) Thread[id=42, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:39081), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
  25) Thread[id=30, name=qtp2024767747-30, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.lang.Thread.run(Thread.java:748)
  26) Thread[id=27, name=qtp2024767747-27, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)
        at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)
        at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
        at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$30/1333528501.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  27) Thread[id=51, name=OverseerAutoScalingTriggerThread-72201792901742595-127.0.0.1:42417_solr-n_0000000000, state=WAITING, group=Overseer autoscaling triggers]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
        at java.lang.Thread.run(Thread.java:748)
  28) Thread[id=47, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:39081), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
  29) Thread[id=58, name=ScheduledTrigger-7-thread-4, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  30) Thread[id=43, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  31) Thread[id=60, name=Thread-11, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)
  32) Thread[id=81, name=qtp2024767747-81, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  33) Thread[id=15, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.lang.Thread.run(Thread.java:748)
  34) Thread[id=69, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[B982107109F46690]-SendThread(127.0.0.1:39081), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
  35) Thread[id=71, name=zkConnectionManagerCallback-29-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  36) Thread[id=49, name=zkConnectionManagerCallback-19-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  37) Thread[id=18, name=ProcessThread(sid:0 cport:39081):, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
  38) Thread[id=83, name=qtp2024767747-83, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)
        at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)
        at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:674)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
        at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.lang.Thread.run(Thread.java:748)
  39) Thread[id=56, name=SolrRrdBackendFactory-11-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  40) Thread[id=79, name=qtp2024767747-79, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1682)
        at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1629)
        at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1744)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1131)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$172/893767298.execute(Unknown Source)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call

[...truncated too long message...]

sponse/transform/package-info.class
    [javac] Creating empty /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/classes/java/org/apache/solr/ltr/search/package-info.class
    [javac] Creating empty /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/classes/java/org/apache/solr/ltr/store/rest/package-info.class
    [javac] Creating empty /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/classes/java/org/apache/solr/ltr/model/package-info.class
    [javac] Creating empty /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/classes/java/org/apache/solr/ltr/store/package-info.class

common-solr.compile-core:

solr-contrib-build.compile-core:

compile-core:

compile-test-framework:

-check-git-state:

-git-cleanroot:

-copy-git-state:

git-autoclean:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/top-level-ivy-settings.xml

resolve:

init:

compile-lucene-core:

compile-codecs:

-check-git-state:

-git-cleanroot:

-copy-git-state:

git-autoclean:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/top-level-ivy-settings.xml

resolve:

common.init:

compile-lucene-core:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

common.compile-core:

compile-core:

common.compile-test:
    [mkdir] Created dir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/classes/test
    [javac] Compiling 38 source files to /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/classes/test
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

compile-test:

BUILD SUCCESSFUL
Total time: 42 seconds
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestLTROnSolrCloud" -Dtests.showOutput=onerror "-Dargs=-XX:+UseCompressedOops -XX:+UseParallelGC" -Dtests.seed=B982107109F46690 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar-SA -Dtests.timezone=America/Campo_Grande -Dtests.asserts=true -Dtests.file.encoding=UTF-8
Buildfile: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/contrib/ltr/build.xml

-clover.disable:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/top-level-ivy-settings.xml

install-junit4-taskdef:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.4.15 in public
[ivy:cachepath] :: resolution report :: resolve 17ms :: artifacts dl 0ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
	---------------------------------------------------------------------

-init-totals:

-test:
    [mkdir] Created dir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test
[junit4:pickseed] Seed property 'tests.seed' already defined: B982107109F46690
    [mkdir] Created dir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/temp
   [junit4] <JUnit4> says ahoj! Master seed: B982107109F46690
   [junit4] Executing 5 suites with 3 JVMs.
   [junit4] 
   [junit4] Started J1 PID(2338@serv1.sd-datasolutions.de).
   [junit4] Started J0 PID(2340@serv1.sd-datasolutions.de).
   [junit4] Started J2 PID(2339@serv1.sd-datasolutions.de).
   [junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud
   [junit4] OK      10.6s J2 | TestLTROnSolrCloud.testSimpleQuery
   [junit4] Completed [1/5] on J2 in 13.07s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud
   [junit4] OK      10.5s J0 | TestLTROnSolrCloud.testSimpleQuery
   [junit4] Completed [2/5] on J0 in 13.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud
   [junit4] OK      10.2s J1 | TestLTROnSolrCloud.testSimpleQuery
   [junit4] Completed [3/5] on J1 in 12.96s, 1 test
   [junit4] 
   [junit4] Duplicate suite name used with XML reports: org.apache.solr.ltr.TestLTROnSolrCloud. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message.
   [junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud
   [junit4] OK      4.43s J2 | TestLTROnSolrCloud.testSimpleQuery
   [junit4] Completed [4/5] on J2 in 4.73s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud
   [junit4] OK      4.48s J1 | TestLTROnSolrCloud.testSimpleQuery
   [junit4] Completed [5/5] on J1 in 4.78s, 1 test
   [junit4] 
   [junit4] JVM J0:     0.39 ..    14.27 =    13.89s
   [junit4] JVM J1:     0.39 ..    19.27 =    18.88s
   [junit4] JVM J2:     0.39 ..    19.01 =    18.63s
   [junit4] Execution time total: 19 seconds
   [junit4] Tests summary: 5 suites, 5 tests
   [junit4] Could not remove temporary path: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/J2 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/J2/temp])
   [junit4] Could not remove temporary path: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/J0 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/J0/temp])
   [junit4] Could not remove temporary path: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/J1 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/J1/temp])
     [echo] 5 slowest tests:
[junit4:tophints]  52.81s | org.apache.solr.ltr.TestLTROnSolrCloud
[junit4:tophints]  17.35s | org.apache.solr.ltr.TestParallelWeightCreation
[junit4:tophints]   5.99s | org.apache.solr.ltr.TestLTRWithFacet
[junit4:tophints]   1.08s | org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments
[junit4:tophints]   1.03s | org.apache.solr.ltr.TestRerankBase

-check-totals:

test-nocompile:

BUILD SUCCESSFUL
Total time: 20 seconds
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.ltr.TestLTROnSolrCloud
[repro] Exiting with code 0
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2845)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2724)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
	at hudson.FilePath.act(FilePath.java:1076)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
	at hudson.model.Build$BuildExecution.post2(Build.java:186)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
	at hudson.model.Run.execute(Run.java:1840)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=61447db8df360dbe69028931efa8b491a127462c, workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #2720
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

Mime
View raw message