lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-12-ea+12) - Build # 22955 - Unstable!
Date Mon, 01 Oct 2018 11:17:12 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22955/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseParallelGC

30 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.ltr.TestLTROnSolrCloud

Error Message:
44 threads leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud:     1) Thread[id=67, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    2) Thread[id=47, name=OverseerStateUpdate-72318894706786307-127.0.0.1:45807_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)         at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    3) Thread[id=48, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    4) Thread[id=73, name=zkConnectionManagerCallback-29-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    5) Thread[id=28, name=qtp1752845885-28, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    6) Thread[id=57, name=ScheduledTrigger-7-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    7) Thread[id=44, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:35753), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)         at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)    8) Thread[id=38, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    9) Thread[id=89, name=SolrRrdBackendFactory-11-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   10) Thread[id=46, name=zkConnectionManagerCallback-12-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   11) Thread[id=74, name=Scheduler-1661882176, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   12) Thread[id=18, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Object.wait(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)   13) Thread[id=27, name=qtp1752845885-27, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   14) Thread[id=56, name=ScheduledTrigger-7-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   15) Thread[id=52, name=OverseerCollectionConfigSetProcessor-72318894706786307-127.0.0.1:45807_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)         at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)         at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   16) Thread[id=81, name=qtp1752845885-81, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   17) Thread[id=33, name=qtp1752845885-33, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2211)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   18) Thread[id=49, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:35753), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)         at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)   19) Thread[id=39, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   20) Thread[id=26, name=qtp1752845885-26, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   21) Thread[id=60, name=SolrRrdBackendFactory-11-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   22) Thread[id=51, name=zkConnectionManagerCallback-19-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   23) Thread[id=82, name=qtp1752845885-82, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   24) Thread[id=40, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   25) Thread[id=50, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   26) Thread[id=45, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   27) Thread[id=83, name=qtp1752845885-83, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   28) Thread[id=17, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   29) Thread[id=61, name=MetricsHistoryHandler-12-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   30) Thread[id=72, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[7E57DC35FC5BEE3D]-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   31) Thread[id=71, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[7E57DC35FC5BEE3D]-SendThread(127.0.0.1:35753), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)         at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)   32) Thread[id=31, name=qtp1752845885-31, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   33) Thread[id=34, name=Scheduler-2030437948, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   34) Thread[id=32, name=qtp1752845885-32, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   35) Thread[id=62, name=Thread-10, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Object.wait(Native Method)         at java.base@12-ea/java.lang.Object.wait(Object.java:328)         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1857)   36) Thread[id=30, name=qtp1752845885-30-acceptor-0@4181670d-ServerConnector@4f46914d{SSL,[ssl, http/1.1]}{127.0.0.1:45807}, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at java.base@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525)         at java.base@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277)         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   37) Thread[id=79, name=qtp1752845885-79, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   38) Thread[id=53, name=OverseerAutoScalingTriggerThread-72318894706786307-127.0.0.1:45807_solr-n_0000000000, state=WAITING, group=Overseer autoscaling triggers]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at app//org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   39) Thread[id=29, name=qtp1752845885-29, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$157/0x0000000100324040.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   40) Thread[id=59, name=ScheduledTrigger-7-thread-4, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   41) Thread[id=16, name=Thread-0, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Object.wait(Native Method)         at java.base@12-ea/java.lang.Thread.join(Thread.java:1308)         at java.base@12-ea/java.lang.Thread.join(Thread.java:1375)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)         at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)         at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)   42) Thread[id=58, name=ScheduledTrigger-7-thread-3, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   43) Thread[id=20, name=ProcessThread(sid:0 cport:35753):, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)   44) Thread[id=75, name=OverseerThreadFactory-9-thread-1, state=WAITING, group=Overseer collection creation process.]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.base@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)         at java.base@12-ea/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 44 threads leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud: 
   1) Thread[id=67, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   2) Thread[id=47, name=OverseerStateUpdate-72318894706786307-127.0.0.1:45807_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
        at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   3) Thread[id=48, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   4) Thread[id=73, name=zkConnectionManagerCallback-29-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   5) Thread[id=28, name=qtp1752845885-28, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   6) Thread[id=57, name=ScheduledTrigger-7-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   7) Thread[id=44, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:35753), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
   8) Thread[id=38, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   9) Thread[id=89, name=SolrRrdBackendFactory-11-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  10) Thread[id=46, name=zkConnectionManagerCallback-12-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  11) Thread[id=74, name=Scheduler-1661882176, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  12) Thread[id=18, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Object.wait(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
  13) Thread[id=27, name=qtp1752845885-27, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  14) Thread[id=56, name=ScheduledTrigger-7-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  15) Thread[id=52, name=OverseerCollectionConfigSetProcessor-72318894706786307-127.0.0.1:45807_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
        at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)
        at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  16) Thread[id=81, name=qtp1752845885-81, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  17) Thread[id=33, name=qtp1752845885-33, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2211)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  18) Thread[id=49, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:35753), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
  19) Thread[id=39, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  20) Thread[id=26, name=qtp1752845885-26, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  21) Thread[id=60, name=SolrRrdBackendFactory-11-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  22) Thread[id=51, name=zkConnectionManagerCallback-19-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  23) Thread[id=82, name=qtp1752845885-82, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  24) Thread[id=40, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  25) Thread[id=50, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  26) Thread[id=45, name=jetty-launcher-1-thread-1-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  27) Thread[id=83, name=qtp1752845885-83, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  28) Thread[id=17, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  29) Thread[id=61, name=MetricsHistoryHandler-12-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  30) Thread[id=72, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[7E57DC35FC5BEE3D]-EventThread, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  31) Thread[id=71, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[7E57DC35FC5BEE3D]-SendThread(127.0.0.1:35753), state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
  32) Thread[id=31, name=qtp1752845885-31, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  33) Thread[id=34, name=Scheduler-2030437948, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  34) Thread[id=32, name=qtp1752845885-32, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  35) Thread[id=62, name=Thread-10, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Object.wait(Native Method)
        at java.base@12-ea/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1857)
  36) Thread[id=30, name=qtp1752845885-30-acceptor-0@4181670d-ServerConnector@4f46914d{SSL,[ssl, http/1.1]}{127.0.0.1:45807}, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525)
        at java.base@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  37) Thread[id=79, name=qtp1752845885-79, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  38) Thread[id=53, name=OverseerAutoScalingTriggerThread-72318894706786307-127.0.0.1:45807_solr-n_0000000000, state=WAITING, group=Overseer autoscaling triggers]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at app//org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  39) Thread[id=29, name=qtp1752845885-29, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$157/0x0000000100324040.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  40) Thread[id=59, name=ScheduledTrigger-7-thread-4, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  41) Thread[id=16, name=Thread-0, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/java.lang.Object.wait(Native Method)
        at java.base@12-ea/java.lang.Thread.join(Thread.java:1308)
        at java.base@12-ea/java.lang.Thread.join(Thread.java:1375)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
        at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)
        at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)
  42) Thread[id=58, name=ScheduledTrigger-7-thread-3, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
  43) Thread[id=20, name=ProcessThread(sid:0 cport:35753):, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
  44) Thread[id=75, name=OverseerThreadFactory-9-thread-1, state=WAITING, group=Overseer collection creation process.]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.base@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@12-ea/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
	at __randomizedtesting.SeedInfo.seed([7E57DC35FC5BEE3D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.ltr.TestLTROnSolrCloud

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=73, name=zkConnectionManagerCallback-29-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    2) Thread[id=17, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPoll.wait(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)         at app//org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    3) Thread[id=61, name=MetricsHistoryHandler-12-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    4) Thread[id=44, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:35753), state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    5) Thread[id=71, name=TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[7E57DC35FC5BEE3D]-SendThread(127.0.0.1:35753), state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    6) Thread[id=89, name=SolrRrdBackendFactory-11-thread-2, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    7) Thread[id=46, name=zkConnectionManagerCallback-12-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    8) Thread[id=74, name=Scheduler-1661882176, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)    9) Thread[id=31, name=qtp1752845885-31, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:329)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   10) Thread[id=34, name=Scheduler-2030437948, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   11) Thread[id=32, name=qtp1752845885-32, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:113)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:146)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:401)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   12) Thread[id=62, name=Thread-10, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Object.wait(Native Method)         at java.base@12-ea/java.lang.Object.wait(Object.java:328)         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1857)   13) Thread[id=79, name=qtp1752845885-79, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/sun.nio.ch.IOUtil.write1(Native Method)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:254)         at java.base@12-ea/java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.base@12-ea/java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at java.base@12-ea/sun.nio.ch.SelectorImpl.begin(SelectorImpl.java:94)         at java.base@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:116)         at java.base@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)         at java.base@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   14) Thread[id=29, name=qtp1752845885-29, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)         at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$157/0x0000000100324040.run(Unknown Source)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   15) Thread[id=33, name=qtp1752845885-33, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2211)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   16) Thread[id=49, name=jetty-launcher-1-thread-1-SendThread(127.0.0.1:35753), state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   17) Thread[id=60, name=SolrRrdBackendFactory-11-thread-1, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)         at java.base@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)   18) Thread[id=51, name=zkConnectionManagerCallback-19-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]         at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)         at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@12-ea/java.lang.Thread.run(Thread.java:835)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=73, name=zkConnectionManagerCallback-29-thread-1, state=WAITING, group=TGRP-TestLTROnSolrCloud]
        at java.base@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@12-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@12-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
   2) Thread[id=17, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RU

[...truncated too long message...]

 s:shard1 r:core_node5 x:collection52_shard1_replica_n2] o.a.s.c.SolrCore [collection52_shard1_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@3bd3d54a
   [junit4]   2> 606104 INFO  (coreCloseExecutor-1264-thread-1) [n:127.0.0.1:40791_solr c:collection52 s:shard1 r:core_node5 x:collection52_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection52.shard1.replica_n2, tag=3bd3d54a
   [junit4]   2> 606105 INFO  (coreCloseExecutor-1264-thread-1) [n:127.0.0.1:40791_solr c:collection52 s:shard1 r:core_node5 x:collection52_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5486a8c1: rootName = solr_40791, domain = solr.core.collection52.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.collection52.shard1.replica_n2 / com.codahale.metrics.MetricRegistry@3e775f0e
   [junit4]   2> 606113 INFO  (jetty-closer-990-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@41fb83a7{/solr,null,UNAVAILABLE}
   [junit4]   2> 606114 INFO  (jetty-closer-990-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 606121 INFO  (coreCloseExecutor-1263-thread-2) [n:127.0.0.1:43199_solr c:collection52 s:shard3 r:core_node12 x:collection52_shard3_replica_n10] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection52.shard3.leader, tag=70fda9c3
   [junit4]   2> 606124 INFO  (coreCloseExecutor-1264-thread-1) [n:127.0.0.1:40791_solr c:collection52 s:shard1 r:core_node5 x:collection52_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection52.shard1.leader, tag=3bd3d54a
   [junit4]   2> 606134 INFO  (jetty-closer-990-thread-2) [    ] o.a.s.c.Overseer Overseer (id=72318944247349253-127.0.0.1:40791_solr-n_0000000000) closing
   [junit4]   2> 606134 INFO  (OverseerStateUpdate-72318944247349253-127.0.0.1:40791_solr-n_0000000000) [n:127.0.0.1:40791_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40791_solr
   [junit4]   2> 606134 WARN  (OverseerAutoScalingTriggerThread-72318944247349253-127.0.0.1:40791_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 606135 INFO  (jetty-closer-990-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@554f411f{/solr,null,UNAVAILABLE}
   [junit4]   2> 606136 INFO  (jetty-closer-990-thread-3) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 606139 INFO  (jetty-closer-990-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@60f14550{/solr,null,UNAVAILABLE}
   [junit4]   2> 606139 INFO  (jetty-closer-990-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 606139 ERROR (SUITE-CustomHighlightComponentTest-seed#[BA8979A146D3853E]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 606140 INFO  (SUITE-CustomHighlightComponentTest-seed#[BA8979A146D3853E]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37671 37671
   [junit4]   2> 606245 INFO  (Thread-583) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37671 37671
   [junit4]   2> 606246 WARN  (Thread-583) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/aliases.json
   [junit4]   2> 	6	/solr/configs/conf
   [junit4]   2> 	5	/solr/collections/collection52/terms/shard1
   [junit4]   2> 	5	/solr/collections/collection52/terms/shard3
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 	2	/solr/collections/collection52/terms/shard2
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/clusterstate.json
   [junit4]   2> 	7	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/collections/collection52/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/72318944247349253-127.0.0.1:40791_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/live_nodes
   [junit4]   2> 	7	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.CustomHighlightComponentTest_BA8979A146D3853E-001
   [junit4]   2> Oct 01, 2018 11:10:22 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=549, maxMBSortInHeap=6.3068565278214255, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@37e2bea7), locale=bo-IN, timezone=America/Moncton
   [junit4]   2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 12-ea (64-bit)/cpus=8,threads=1,free=131058816,total=515899392
   [junit4]   2> NOTE: All tests run in this JVM: [IndexSizeTriggerTest, DistributedMLTComponentTest, CustomHighlightComponentTest, CustomHighlightComponentTest, CdcrBootstrapTest, SchemaApiFailureTest, SchemaApiFailureTest, SchemaApiFailureTest, CdcrBootstrapTest, CdcrBootstrapTest, DistributedMLTComponentTest, DistributedMLTComponentTest, CustomHighlightComponentTest]
   [junit4] Completed [23/25 (11!)] on J1 in 13.41s, 1 test, 1 error <<< FAILURES!
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.cdcr.CdcrBootstrapTest
   [junit4] OK      18.2s J0 | CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap
   [junit4] OK      16.7s J0 | CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster
   [junit4] OK      12.9s J0 | CdcrBootstrapTest.testBootstrapWithSourceCluster
   [junit4] Completed [24/25 (11!)] on J0 in 47.79s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.handler.component.CustomHighlightComponentTest
   [junit4] OK      0.58s J2 | CustomHighlightComponentTest.test
   [junit4] Completed [25/25 (11!)] on J2 in 5.79s, 1 test
   [junit4] 
   [junit4] 
   [junit4] Tests with failures [seed: BA8979A146D3853E] (first 10 out of 14):
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration
   [junit4]   - org.apache.solr.handler.component.DistributedMLTComponentTest.test
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration
   [junit4]   - org.apache.solr.schema.SchemaApiFailureTest (suite)
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds
   [junit4]   - org.apache.solr.schema.SchemaApiFailureTest (suite)
   [junit4]   - org.apache.solr.handler.component.DistributedMLTComponentTest.test
   [junit4]   - org.apache.solr.handler.component.CustomHighlightComponentTest (suite)
   [junit4]   - org.apache.solr.cloud.cdcr.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     0.83 ..   610.29 =   609.46s
   [junit4] JVM J1:     0.79 ..   607.35 =   606.56s
   [junit4] JVM J2:     0.80 ..   612.39 =   611.59s
   [junit4] Execution time total: 10 minutes 12 seconds
   [junit4] Tests summary: 25 suites, 47 tests, 3 suite-level errors, 5 errors, 6 failures

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1092: There were test failures: 25 suites, 47 tests, 3 suite-level errors, 5 errors, 6 failures [seed: BA8979A146D3853E]

Total time: 10 minutes 14 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   2/5 failed: org.apache.solr.cloud.cdcr.CdcrBootstrapTest
[repro]   2/5 failed: org.apache.solr.handler.component.CustomHighlightComponentTest
[repro]   2/5 failed: org.apache.solr.handler.component.DistributedMLTComponentTest
[repro]   2/5 failed: org.apache.solr.schema.SchemaApiFailureTest
[repro]   3/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro]   3/5 failed: org.apache.solr.ltr.TestLTROnSolrCloud
[repro]   4/5 failed: org.apache.solr.client.solrj.impl.CloudSolrClientTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=93e2ae3fc454f1f376a7b4389905ebeb1732af70, workspace=/var/lib/jenkins/workspace/Lucene-Solr-master-Linux
[WARNINGS] Computing warning deltas based on reference build #22954
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

Mime
View raw message