lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 116 - Failure!
Date Thu, 17 Aug 2017 15:56:36 GMT
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/116/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

No tests ran.

Build Log:
[...truncated 11567 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 1520249 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/init-core-data-001
   [junit4]   2> 1520249 WARN  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 1520267 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1520274 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1520282 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001
   [junit4]   2> 1520286 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1520311 INFO  (Thread-4650) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1520311 INFO  (Thread-4650) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1520382 ERROR (Thread-4650) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1520409 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:33834
   [junit4]   2> 1521190 INFO  (jetty-launcher-1286-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1521190 INFO  (jetty-launcher-1286-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1521191 INFO  (jetty-launcher-1286-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1521192 INFO  (jetty-launcher-1286-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1521204 INFO  (jetty-launcher-1286-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@65dd9a85{/solr,null,AVAILABLE}
   [junit4]   2> 1521212 INFO  (jetty-launcher-1286-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2a91d163{/solr,null,AVAILABLE}
   [junit4]   2> 1521222 INFO  (jetty-launcher-1286-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@77739e8e{/solr,null,AVAILABLE}
   [junit4]   2> 1521227 INFO  (jetty-launcher-1286-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2ec574f3{HTTP/1.1,[http/1.1]}{127.0.0.1:38253}
   [junit4]   2> 1521227 INFO  (jetty-launcher-1286-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11b84194{HTTP/1.1,[http/1.1]}{127.0.0.1:55541}
   [junit4]   2> 1521238 INFO  (jetty-launcher-1286-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@18fb9768{HTTP/1.1,[http/1.1]}{127.0.0.1:34334}
   [junit4]   2> 1521239 INFO  (jetty-launcher-1286-thread-1) [    ] o.e.j.s.Server Started @1527404ms
   [junit4]   2> 1521239 INFO  (jetty-launcher-1286-thread-3) [    ] o.e.j.s.Server Started @1527404ms
   [junit4]   2> 1521239 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38253}
   [junit4]   2> 1521239 INFO  (jetty-launcher-1286-thread-4) [    ] o.e.j.s.Server Started @1527405ms
   [junit4]   2> 1521239 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34334}
   [junit4]   2> 1521240 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55541}
   [junit4]   2> 1521265 INFO  (jetty-launcher-1286-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@338652c0{/solr,null,AVAILABLE}
   [junit4]   2> 1521266 ERROR (jetty-launcher-1286-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1521266 ERROR (jetty-launcher-1286-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1521272 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1521272 ERROR (jetty-launcher-1286-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1521272 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1521272 INFO  (jetty-launcher-1286-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33076f8{HTTP/1.1,[http/1.1]}{127.0.0.1:65497}
   [junit4]   2> 1521279 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1521278 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1521280 INFO  (jetty-launcher-1286-thread-2) [    ] o.e.j.s.Server Started @1527446ms
   [junit4]   2> 1521280 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1521280 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1521280 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1521281 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65497}
   [junit4]   2> 1521281 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1521281 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-08-15T16:57:42.653Z
   [junit4]   2> 1521281 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-08-15T16:57:42.653Z
   [junit4]   2> 1521281 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1521283 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-08-15T16:57:42.655Z
   [junit4]   2> 1521282 ERROR (jetty-launcher-1286-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1521283 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1521283 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1521284 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1521284 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-08-15T16:57:42.656Z
   [junit4]   2> 1521852 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1521858 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1521917 INFO  (jetty-launcher-1286-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33834/solr
   [junit4]   2> 1521919 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1521920 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1521930 INFO  (jetty-launcher-1286-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33834/solr
   [junit4]   2> 1522574 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15de6d456450003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1522606 INFO  (jetty-launcher-1286-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33834/solr
   [junit4]   2> 1522606 INFO  (jetty-launcher-1286-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33834/solr
   [junit4]   2> 1522606 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15de6d456450004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1524122 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1524126 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55541_solr
   [junit4]   2> 1524138 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.c.Overseer Overseer (id=98488566526836748-127.0.0.1:55541_solr-n_0000000000) starting
   [junit4]   2> 1524278 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55541_solr
   [junit4]   2> 1524290 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1524386 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1524406 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1524425 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38253_solr
   [junit4]   2> 1524428 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1524460 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1524483 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1524539 INFO  (zkCallback-1307-thread-1-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1524545 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1524558 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65497_solr
   [junit4]   2> 1524565 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1524565 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1524566 INFO  (zkCallback-1307-thread-1-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1524915 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1525227 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1525230 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34334_solr
   [junit4]   2> 1525242 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1525412 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1525242 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1525242 INFO  (zkCallback-1307-thread-1-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1526464 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38253.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527174 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65497.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527246 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34334.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527248 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38253.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527248 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38253.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527257 INFO  (jetty-launcher-1286-thread-1) [n:127.0.0.1:38253_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node1/.
   [junit4]   2> 1527277 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55541.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527293 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34334.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527293 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34334.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527302 INFO  (jetty-launcher-1286-thread-3) [n:127.0.0.1:34334_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node3/.
   [junit4]   2> 1527303 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65497.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527304 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65497.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527309 INFO  (jetty-launcher-1286-thread-2) [n:127.0.0.1:65497_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node2/.
   [junit4]   2> 1527742 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55541.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527743 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55541.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1527745 INFO  (jetty-launcher-1286-thread-4) [n:127.0.0.1:55541_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node4/.
   [junit4]   2> 1528659 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1528660 INFO  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33834/solr ready
   [junit4]   2> 1529443 INFO  (TEST-DeleteReplicaTest.deleteReplicaByCount-seed#[66D0ABDB1709B333]) [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteReplicaByCount
   [junit4]   2> 1529985 INFO  (qtp792472765-11472) [n:127.0.0.1:65497_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=2&replicationFactor=0&collection.configName=conf&name=deleteByCount&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1529995 INFO  (OverseerThreadFactory-2778-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.CreateCollectionCmd Create collection deleteByCount
   [junit4]   2> 1530664 INFO  (OverseerStateUpdate-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCount",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCount_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55541/solr",
   [junit4]   2>   "type":"TLOG"} 
   [junit4]   2> 1530674 INFO  (OverseerStateUpdate-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCount",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCount_shard1_replica_p2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38253/solr",
   [junit4]   2>   "type":"PULL"} 
   [junit4]   2> 1530687 INFO  (OverseerStateUpdate-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCount",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCount_shard1_replica_p4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34334/solr",
   [junit4]   2>   "type":"PULL"} 
   [junit4]   2> 1531401 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_p4&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1531401 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_t1&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 1531401 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_p2&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1531404 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1531404 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1531405 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1531552 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1531552 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1531552 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1531648 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1531649 INFO  (zkCallback-1299-thread-2-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1531649 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1533444 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1533444 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1533496 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1533644 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.s.IndexSchema [deleteByCount_shard1_replica_p2] Schema name=minimal
   [junit4]   2> 1533644 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.s.IndexSchema [deleteByCount_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 1533658 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1533658 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCount_shard1_replica_p2' using configuration from collection deleteByCount, trusted=true
   [junit4]   2> 1533659 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1533659 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCount_shard1_replica_t1' using configuration from collection deleteByCount, trusted=true
   [junit4]   2> 1533660 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.IndexSchema [deleteByCount_shard1_replica_p4] Schema name=minimal
   [junit4]   2> 1533673 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1533673 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCount_shard1_replica_p4' using configuration from collection deleteByCount, trusted=true
   [junit4]   2> 1533890 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34334.solr.core.deleteByCount.shard1.replica_p4' (registry 'solr.core.deleteByCount.shard1.replica_p4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1533890 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38253.solr.core.deleteByCount.shard1.replica_p2' (registry 'solr.core.deleteByCount.shard1.replica_p2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1533898 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55541.solr.core.deleteByCount.shard1.replica_t1' (registry 'solr.core.deleteByCount.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1533901 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1533901 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1533904 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1533906 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrCore [[deleteByCount_shard1_replica_p4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node3/deleteByCount_shard1_replica_p4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node3/./deleteByCount_shard1_replica_p4/data/]
   [junit4]   2> 1533907 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.SolrCore [[deleteByCount_shard1_replica_p2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node1/deleteByCount_shard1_replica_p2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node1/./deleteByCount_shard1_replica_p2/data/]
   [junit4]   2> 1533909 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SolrCore [[deleteByCount_shard1_replica_t1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node4/deleteByCount_shard1_replica_t1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node4/./deleteByCount_shard1_replica_t1/data/]
   [junit4]   2> 1534738 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1534738 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1534741 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6d7b44c[deleteByCount_shard1_replica_p2] main]
   [junit4]   2> 1534743 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1535146 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1535197 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1535203 INFO  (searcherExecutor-2787-thread-1-processing-n:127.0.0.1:38253_solr x:deleteByCount_shard1_replica_p2 s:shard1 c:deleteByCount r:core_node5) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p2] Registered new searcher Searcher@6d7b44c[deleteByCount_shard1_replica_p2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1535203 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.ZkController deleteByCount_shard1_replica_p2 starting background replication from leader
   [junit4]   2> 1535207 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 1535213 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 1535222 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1535222 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1535225 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.SolrIndexSearcher Opening [Searcher@3ef1686e[deleteByCount_shard1_replica_p4] main]
   [junit4]   2> 1535229 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1535229 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1535229 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1535230 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1535231 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1535232 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.ZkController deleteByCount_shard1_replica_p4 starting background replication from leader
   [junit4]   2> 1535232 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1535232 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1535232 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 1535234 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@39647d0d[deleteByCount_shard1_replica_t1] main]
   [junit4]   2> 1535235 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 1535241 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1535245 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1535245 INFO  (searcherExecutor-2789-thread-1-processing-n:127.0.0.1:34334_solr x:deleteByCount_shard1_replica_p4 s:shard1 c:deleteByCount r:core_node6) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p4] Registered new searcher Searcher@3ef1686e[deleteByCount_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1535246 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1535248 INFO  (searcherExecutor-2788-thread-1-processing-n:127.0.0.1:55541_solr x:deleteByCount_shard1_replica_t1 s:shard1 c:deleteByCount r:core_node3) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SolrCore [deleteByCount_shard1_replica_t1] Registered new searcher Searcher@39647d0d[deleteByCount_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1535249 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1575817080074141696
   [junit4]   2> 1535417 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1535418 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1535418 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55541/solr/deleteByCount_shard1_replica_t1/
   [junit4]   2> 1535418 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1535418 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.SyncStrategy http://127.0.0.1:55541/solr/deleteByCount_shard1_replica_t1/ has no replicas
   [junit4]   2> 1535418 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1535419 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.ZkController deleteByCount_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 1535425 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1535425 INFO  (zkCallback-1299-thread-2-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1535425 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1535425 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1535425 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1535426 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1535744 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55541/solr/deleteByCount_shard1_replica_t1/ shard1
   [junit4]   2> 1536031 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536031 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536032 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536032 INFO  (zkCallback-1299-thread-2-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536032 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536033 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536076 INFO  (indexFetcher-2803-thread-1) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.h.IndexFetcher Replica core_node3 is leader but it's state is down, skipping replication
   [junit4]   2> 1536076 INFO  (indexFetcher-2796-thread-1) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.h.IndexFetcher Replica core_node3 is leader but it's state is down, skipping replication
   [junit4]   2> 1536180 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1536199 INFO  (qtp528659115-11462) [n:127.0.0.1:55541_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_t1&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=4800
   [junit4]   2> 1536221 INFO  (qtp164563617-11482) [n:127.0.0.1:38253_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_p2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_p2&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=4822
   [junit4]   2> 1536253 INFO  (qtp741588902-11474) [n:127.0.0.1:34334_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_p4&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=4855
   [junit4]   2> 1536357 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536357 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536357 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536357 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536357 INFO  (zkCallback-1299-thread-2-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536357 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1536413 INFO  (qtp792472765-11472) [n:127.0.0.1:65497_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1536414 INFO  (qtp792472765-11472) [n:127.0.0.1:65497_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=2&replicationFactor=0&collection.configName=conf&name=deleteByCount&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2} status=0 QTime=6434
   [junit4]   2> 1536424 INFO  (qtp792472765-11489) [n:127.0.0.1:65497_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deletereplica with params count=2&action=DELETEREPLICA&collection=deleteByCount&shard=shard1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1536437 INFO  (OverseerCollectionConfigSetProcessor-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1536657 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCount.shard1.replica_p4, tag=null
   [junit4]   2> 1536897 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@ffa5cc5: rootName = solr_34334, domain = solr.core.deleteByCount.shard1.replica_p4, service url = null, agent id = null] for registry solr.core.deleteByCount.shard1.replica_p4 / com.codahale.metrics.MetricRegistry@1d8254cb
   [junit4]   2> 1536923 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.c.ZkController deleteByCount_shard1_replica_p4 stopping background replication from leader
   [junit4]   2> 1537150 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p4]  CLOSING SolrCore org.apache.solr.core.SolrCore@24d21201
   [junit4]   2> 1537150 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCount.shard1.replica_p4, tag=617746945
   [junit4]   2> 1537150 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteByCount.shard1.leader, tag=617746945
   [junit4]   2> 1537162 INFO  (qtp741588902-11475) [n:127.0.0.1:34334_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=deleteByCount_shard1_replica_p4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2&deleteIndex=true} status=0 QTime=516
   [junit4]   2> 1537278 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1537278 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1537278 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1537278 INFO  (zkCallback-1299-thread-2-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1537422 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCount.shard1.replica_p2, tag=null
   [junit4]   2> 1537423 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@17deb455: rootName = solr_38253, domain = solr.core.deleteByCount.shard1.replica_p2, service url = null, agent id = null] for registry solr.core.deleteByCount.shard1.replica_p2 / com.codahale.metrics.MetricRegistry@2c48a999
   [junit4]   2> 1537557 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.c.ZkController deleteByCount_shard1_replica_p2 stopping background replication from leader
   [junit4]   2> 1537563 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p2]  CLOSING SolrCore org.apache.solr.core.SolrCore@781dd184
   [junit4]   2> 1537563 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCount.shard1.replica_p2, tag=2015220100
   [junit4]   2> 1537563 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteByCount.shard1.leader, tag=2015220100
   [junit4]   2> 1537654 INFO  (qtp164563617-11483) [n:127.0.0.1:38253_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=deleteByCount_shard1_replica_p2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2&deleteIndex=true} status=0 QTime=267
   [junit4]   2> 1537762 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1537762 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1537921 INFO  (qtp792472765-11489) [n:127.0.0.1:65497_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={count=2&action=DELETEREPLICA&collection=deleteByCount&shard=shard1&wt=javabin&version=2} status=0 QTime=1497
   [junit4]   2> 1537925 INFO  (qtp792472765-11490) [n:127.0.0.1:65497_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deletereplica with params count=1&action=DELETEREPLICA&collection=deleteByCount&shard=shard1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1537929 ERROR (OverseerThreadFactory-2778-thread-3-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.OverseerCollectionMessageHandler Collection: deleteByCount operation: deletereplica failed:org.apache.solr.common.SolrException: There is only one replica available in shard/collection: shard1/deleteByCount. Cannot delete that.
   [junit4]   2> 	at org.apache.solr.cloud.DeleteReplicaCmd.validateReplicaAvailability(DeleteReplicaCmd.java:192)
   [junit4]   2> 	at org.apache.solr.cloud.DeleteReplicaCmd.pickReplicasTobeDeleted(DeleteReplicaCmd.java:159)
   [junit4]   2> 	at org.apache.solr.cloud.DeleteReplicaCmd.deleteReplicaBasedOnCount(DeleteReplicaCmd.java:128)
   [junit4]   2> 	at org.apache.solr.cloud.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:79)
   [junit4]   2> 	at org.apache.solr.cloud.DeleteReplicaCmd.call(DeleteReplicaCmd.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:227)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1537941 INFO  (OverseerCollectionConfigSetProcessor-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1538153 ERROR (OverseerCollectionConfigSetProcessor-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.OverseerTaskProcessor :org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000004
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:374)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:371)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:371)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:572)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskQueue.remove(OverseerTaskQueue.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.cleanUpWorkQueue(OverseerTaskProcessor.java:321)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:202)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1538154 INFO  (qtp792472765-11490) [n:127.0.0.1:65497_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={count=1&action=DELETEREPLICA&collection=deleteByCount&shard=shard1&wt=javabin&version=2} status=400 QTime=229
   [junit4]   2> 1538155 INFO  (OverseerCollectionConfigSetProcessor-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1538157 INFO  (TEST-DeleteReplicaTest.deleteReplicaByCount-seed#[66D0ABDB1709B333]) [    ] o.a.s.SolrTestCaseJ4 ###Ending deleteReplicaByCount
   [junit4]   2> 1538375 INFO  (TEST-DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup-seed#[66D0ABDB1709B333]) [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteReplicaAndVerifyDirectoryCleanup
   [junit4]   2> 1538376 INFO  (qtp792472765-11490) [n:127.0.0.1:65497_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&name=deletereplica_test&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1538379 INFO  (OverseerThreadFactory-2778-thread-4-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.CreateCollectionCmd Create collection deletereplica_test
   [junit4]   2> 1538604 INFO  (OverseerStateUpdate-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deletereplica_test",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deletereplica_test_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34334/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 1538659 INFO  (OverseerStateUpdate-98488566526836748-127.0.0.1:55541_solr-n_0000000000) [n:127.0.0.1:55541_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deletereplica_test",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deletereplica_test_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55541/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 1538896 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=deletereplica_test_shard1_replica_n1&action=CREATE&numShards=1&collection=deletereplica_test&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1538897 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=deletereplica_test_shard1_replica_n3&action=CREATE&numShards=1&collection=deletereplica_test&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1539040 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1539040 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1539041 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1539041 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1539935 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1540733 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1540734 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.s.IndexSchema [deletereplica_test_shard1_replica_n3] Schema name=minimal
   [junit4]   2> 1540903 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1540903 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'deletereplica_test_shard1_replica_n3' using configuration from collection deletereplica_test, trusted=true
   [junit4]   2> 1540903 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55541.solr.core.deletereplica_test.shard1.replica_n3' (registry 'solr.core.deletereplica_test.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1540903 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1540904 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.SolrCore [[deletereplica_test_shard1_replica_n3] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node4/deletereplica_test_shard1_replica_n3], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node4/./deletereplica_test_shard1_replica_n3/data/]
   [junit4]   2> 1540935 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.s.IndexSchema [deletereplica_test_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1540951 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1540951 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'deletereplica_test_shard1_replica_n1' using configuration from collection deletereplica_test, trusted=true
   [junit4]   2> 1540952 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34334.solr.core.deletereplica_test.shard1.replica_n1' (registry 'solr.core.deletereplica_test.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d5f14a
   [junit4]   2> 1540952 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1540955 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrCore [[deletereplica_test_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node3/deletereplica_test_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_66D0ABDB1709B333-001/tempDir-001/node3/./deletereplica_test_shard1_replica_n1/data/]
   [junit4]   2> 1541181 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1541181 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1541185 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1541185 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1541186 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@6451cf02[deletereplica_test_shard1_replica_n3] main]
   [junit4]   2> 1541188 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1541190 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1541190 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1541191 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1575817086304780288
   [junit4]   2> 1541205 INFO  (searcherExecutor-2804-thread-1-processing-n:127.0.0.1:55541_solr x:deletereplica_test_shard1_replica_n3 s:shard1 c:deletereplica_test r:core_node4) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.SolrCore [deletereplica_test_shard1_replica_n3] Registered new searcher Searcher@6451cf02[deletereplica_test_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1541227 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1541233 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1541233 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1541233 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1541234 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1541415 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1541416 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1541768 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1541769 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1541772 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@34d76838[deletereplica_test_shard1_replica_n1] main]
   [junit4]   2> 1541775 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1541775 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1541776 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1541778 INFO  (searcherExecutor-2805-thread-1-processing-n:127.0.0.1:34334_solr x:deletereplica_test_shard1_replica_n1 s:shard1 c:deletereplica_test r:core_node2) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrCore [deletereplica_test_shard1_replica_n1] Registered new searcher Searcher@34d76838[deletereplica_test_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1541779 INFO  (qtp741588902-11476) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1575817086921342976
   [junit4]   2> 1542271 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1542272 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1542272 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55541/solr/deletereplica_test_shard1_replica_n3/
   [junit4]   2> 1542272 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.PeerSync PeerSync: core=deletereplica_test_shard1_replica_n3 url=http://127.0.0.1:55541/solr START replicas=[http://127.0.0.1:34334/solr/deletereplica_test_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 1542574 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.u.PeerSync PeerSync: core=deletereplica_test_shard1_replica_n3 url=http://127.0.0.1:55541/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1558636 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 1558636 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 1568458 WARN  (jetty-launcher-1286-thread-2-SendThread(127.0.0.1:33834)) [n:127.0.0.1:65497_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 30052ms for sessionid 0x15de6d45645000b
   [junit4]   2> 1568462 WARN  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker-SendThread(127.0.0.1:33834)) [    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 30049ms for sessionid 0x15de6d45645000e
   [junit4]   2> 1568473 WARN  (jetty-launcher-1286-thread-1-SendThread(127.0.0.1:33834)) [n:127.0.0.1:38253_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 30050ms for sessionid 0x15de6d456450007
   [junit4]   2> 1570191 WARN  (SUITE-DeleteReplicaTest-seed#[66D0ABDB1709B333]-worker-SendThread(127.0.0.1:33834)) [    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15de6d45645000e has expired
   [junit4]   2> 1570292 WARN  (jetty-launcher-1286-thread-1-SendThread(127.0.0.1:33834)) [n:127.0.0.1:38253_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15de6d456450007 has expired
   [junit4]   2> 1570558 WARN  (jetty-launcher-1286-thread-2-SendThread(127.0.0.1:33834)) [n:127.0.0.1:65497_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15de6d45645000b has expired
   [junit4]   2> 1577759 WARN  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3dc2b206 name: ZooKeeperConnection Watcher:127.0.0.1:33834/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1577765 WARN  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1577784 INFO  (qtp741588902-11478) [n:127.0.0.1:34334_solr c:deletereplica_test s:shard1 r:core_node2 x:deletereplica_test_shard1_replica_n1] o.a.s.c.S.Request [deletereplica_test_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=20
   [junit4]   2> 1577837 WARN  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@ef4fb8a name: ZooKeeperConnection Watcher:127.0.0.1:33834/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 1577839 WARN  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 1577845 WARN  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 1577842 WARN  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ca8e0e4 name: ZooKeeperConnection Watcher:127.0.0.1:33834/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 1577848 WARN  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 1577849 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1577849 WARN  (zkCallback-1316-thread-5) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@ef4fb8a name: ZooKeeperConnection Watcher:127.0.0.1:33834/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1577853 WARN  (zkCallback-1316-thread-5) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1577857 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1577858 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1577859 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1577839 WARN  (zkCallback-1307-thread-3-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ca8e0e4 name: ZooKeeperConnection Watcher:127.0.0.1:33834/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1577865 WARN  (zkCallback-1307-thread-3-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1577850 WARN  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 1577922 WARN  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3dc2b206 name: ZooKeeperConnection Watcher:127.0.0.1:33834/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 1577924 WARN  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 1577926 WARN  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 1577953 INFO  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 1577956 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 1577958 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 1577970 INFO  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 1577971 INFO  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 1577989 INFO  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 1577990 INFO  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 1577995 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 1578014 INFO  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 1578015 INFO  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 1578037 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55541/solr/deletereplica_test_shard1_replica_n3/ shard1
   [junit4]   2> 1578057 INFO  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38253_solr
   [junit4]   2> 1578062 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65497_solr
   [junit4]   2> 1578065 INFO  (zkCallback-1308-thread-1-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1578066 INFO  (zkCallback-1308-thread-2-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1578067 INFO  (zkCallback-1306-thread-2-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1578068 INFO  (zkCallback-1306-thread-1-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1578077 INFO  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1578077 INFO  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 1578081 INFO  (zkCallback-1299-thread-3-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 1578081 INFO  (zkCallback-1299-thread-1-processing-n:127.0.0.1:38253_solr) [n:127.0.0.1:38253_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 1578102 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 1578104 INFO  (zkCallback-1307-thread-2-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 1578094 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1578086 INFO  (zkCallback-1307-thread-1-processing-n:127.0.0.1:65497_solr) [n:127.0.0.1:65497_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 1578085 INFO  (zkCallback-1306-thread-3-processing-n:127.0.0.1:34334_solr) [n:127.0.0.1:34334_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 1578104 INFO  (zkCallback-1308-thread-3-processing-n:127.0.0.1:55541_solr) [n:127.0.0.1:55541_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 1578104 INFO  (zkCallback-1316-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1578136 INFO  (qtp528659115-11463) [n:127.0.0.1:55541_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplic

[...truncated too long message...]

Thread.java:748)
   [junit4]    >    9) Thread[id=14455, name=qtp599841379-14455, state=RUNNABLE, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   10) Thread[id=14459, name=qtp599841379-14459, state=TIMED_WAITING, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   11) Thread[id=14464, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   12) Thread[id=14456, name=qtp599841379-14456-acceptor-0@32820166-ServerConnector@19f6bcd3{HTTP/1.1,[http/1.1]}{127.0.0.1:65008}, state=RUNNABLE, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
   [junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
   [junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
   [junit4]    >         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
   [junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   13) Thread[id=14460, name=qtp599841379-14460, state=TIMED_WAITING, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([66D0ABDB1709B333]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=14467, name=Thread-5989, state=WAITING, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:502)
   [junit4]    >         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1617)
   [junit4]    >    2) Thread[id=14472, name=Scheduler-1451936798, state=WAITING, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    3) Thread[id=14455, name=qtp599841379-14455, state=RUNNABLE, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=14463, name=org.eclipse.jetty.server.session.HashSessionManager@1d7d26e3Timer, state=TIMED_WAITING, group=TGRP-CoreAdminHandlerTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([66D0ABDB1709B333]:0)
   [junit4] Completed [306/729 (4!)] on J0 in 674.75s, 6 tests, 3 errors <<< FAILURES!

[...truncated 1431 lines...]
ERROR: command execution failed.
ERROR: Step ‘Archive the artifacts’ failed: no workspace for Lucene-Solr-7.x-Solaris #116
ERROR: Step ‘Scan for compiler warnings’ failed: no workspace for Lucene-Solr-7.x-Solaris #116
ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for Lucene-Solr-7.x-Solaris #116
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

Mime
View raw message