lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-repro - Build # 2645 - Unstable
Date Tue, 08 Jan 2019 21:14:54 GMT
Build: https://builds.apache.org/job/Lucene-Solr-repro/2645/

[...truncated 28 lines...]
[repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/426/consoleText

[repro] Revision: 5c813f37d34c0e8dc4037ec47db86e795df778cd

[repro] Ant options: -Dtests.multiplier=2 -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
[repro] Repro line:  ant test  -Dtestcase=TestDistributedSearch -Dtests.method=test -Dtests.seed=1B45B21C0A901F9 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=es-PA -Dtests.timezone=US/Arizona -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=1B45B21C0A901F9 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=it -Dtests.timezone=America/Buenos_Aires -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 951b4e4c83756d2d5b8592168cdba828a9133ba3
[repro] git fetch
[repro] git checkout 5c813f37d34c0e8dc4037ec47db86e795df778cd

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestDistributedSearch
[repro]       ForceLeaderTest
[repro] ant compile-test

[...truncated 3605 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.TestDistributedSearch|*.ForceLeaderTest" -Dtests.showOutput=onerror -Dtests.multiplier=2 -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.seed=1B45B21C0A901F9 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=es-PA -Dtests.timezone=US/Arizona -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 1861 lines...]
   [junit4]   2> 261966 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase ERROR: org.apache.solr.common.SolrException: Could not find a healthy node to handle the request. ... Sleeping for 1 seconds before re-try ...
   [junit4]   2> 262509 ERROR (indexFetcher-307-thread-1) [    ] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: forceleader_test_collection slice: shard1 saw state=DocCollection(forceleader_test_collection//collections/forceleader_test_collection/state.json/15)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"0",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t1",
   [junit4]   2>           "base_url":"http://127.0.0.1:37059/k_dpv/c",
   [junit4]   2>           "node_name":"127.0.0.1:37059_k_dpv%2Fc",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"TLOG"},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           "state":"down",
   [junit4]   2>           "base_url":"http://127.0.0.1:34761/k_dpv/c",
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t3",
   [junit4]   2>           "node_name":"127.0.0.1:34761_k_dpv%2Fc",
   [junit4]   2>           "force_set_state":"false",
   [junit4]   2>           "type":"TLOG"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "state":"down",
   [junit4]   2>           "base_url":"http://127.0.0.1:44452/k_dpv/c",
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t5",
   [junit4]   2>           "node_name":"127.0.0.1:44452_k_dpv%2Fc",
   [junit4]   2>           "force_set_state":"false",
   [junit4]   2>           "type":"TLOG"}}}},
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"0",
   [junit4]   2>   "tlogReplicas":"3"} with live_nodes=[127.0.0.1:34761_k_dpv%2Fc, 127.0.0.1:43565_k_dpv%2Fc, 127.0.0.1:44452_k_dpv%2Fc]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:902)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:879)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.getLeaderReplica(IndexFetcher.java:688)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:381)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 262509 INFO  (recoveryExecutor-286-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
   [junit4]   2> 262535 ERROR (indexFetcher-300-thread-1) [    ] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: forceleader_test_collection slice: shard1 saw state=DocCollection(forceleader_test_collection//collections/forceleader_test_collection/state.json/15)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"0",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t1",
   [junit4]   2>           "base_url":"http://127.0.0.1:37059/k_dpv/c",
   [junit4]   2>           "node_name":"127.0.0.1:37059_k_dpv%2Fc",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"TLOG"},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           "state":"down",
   [junit4]   2>           "base_url":"http://127.0.0.1:34761/k_dpv/c",
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t3",
   [junit4]   2>           "node_name":"127.0.0.1:34761_k_dpv%2Fc",
   [junit4]   2>           "force_set_state":"false",
   [junit4]   2>           "type":"TLOG"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "state":"down",
   [junit4]   2>           "base_url":"http://127.0.0.1:44452/k_dpv/c",
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t5",
   [junit4]   2>           "node_name":"127.0.0.1:44452_k_dpv%2Fc",
   [junit4]   2>           "force_set_state":"false",
   [junit4]   2>           "type":"TLOG"}}}},
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"0",
   [junit4]   2>   "tlogReplicas":"3"} with live_nodes=[127.0.0.1:34761_k_dpv%2Fc, 127.0.0.1:43565_k_dpv%2Fc, 127.0.0.1:44452_k_dpv%2Fc]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:902)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:879)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.getLeaderReplica(IndexFetcher.java:688)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:381)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 262535 INFO  (recoveryExecutor-312-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
   [junit4]   2> 262536 INFO  (updateExecutor-311-thread-2-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 262536 INFO  (updateExecutor-311-thread-2-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ActionThrottle Throttling recovery attempts - waiting for 6246ms
   [junit4]   2> 262969 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=0 commError=false errorCode=510 
   [junit4]   2> 262969 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 262969 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 262970 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=1 commError=false errorCode=510 
   [junit4]   2> 262970 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 262970 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 262974 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=2 commError=false errorCode=510 
   [junit4]   2> 262974 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 262974 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 262974 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=3 commError=false errorCode=510 
   [junit4]   2> 262974 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 262975 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 262975 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=4 commError=false errorCode=510 
   [junit4]   2> 262975 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 262975 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 262978 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=5 commError=false errorCode=510 
   [junit4]   2> 262978 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 262978 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase ERROR: org.apache.solr.common.SolrException: Could not find a healthy node to handle the request. ... Sleeping for 1 seconds before re-try ...
   [junit4]   2> 263980 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=0 commError=false errorCode=510 
   [junit4]   2> 263980 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 263980 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 263981 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=1 commError=false errorCode=510 
   [junit4]   2> 263981 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 263981 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 263982 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=2 commError=false errorCode=510 
   [junit4]   2> 263982 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 263982 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 263983 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=3 commError=false errorCode=510 
   [junit4]   2> 263983 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 263983 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 263984 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=4 commError=false errorCode=510 
   [junit4]   2> 263984 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 263984 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 263985 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=5 commError=false errorCode=510 
   [junit4]   2> 263985 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 263985 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase ERROR: org.apache.solr.common.SolrException: Could not find a healthy node to handle the request. ... Sleeping for 1 seconds before re-try ...
   [junit4]   2> 264169 INFO  (qtp2024231405-2588) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n5:INDEX.sizeInBytes&key=solr.core.forceleader_test_collection.shard1.replica_t3:INDEX.sizeInBytes} status=0 QTime=1
   [junit4]   2> 264174 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n5:INDEX.sizeInBytes&key=solr.core.forceleader_test_collection.shard1.replica_t3:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264175 INFO  (qtp2024231405-2591) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n5:INDEX.sizeInBytes&key=solr.core.forceleader_test_collection.shard1.replica_t3:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264177 INFO  (qtp2024231405-2590) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264180 INFO  (qtp2024231405-2592) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264186 INFO  (qtp2024231405-2588) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264187 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/
   [junit4]   2> 264187 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c START replicas=[http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/] nUpdates=100
   [junit4]   2> 264188 INFO  (SocketProxy-Acceptor-34761) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57782,localport=34761], receiveBufferSize:531000
   [junit4]   2> 264190 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264192 INFO  (qtp2024231405-2590) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264193 INFO  (qtp2024231405-2590) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 264193 INFO  (SocketProxy-Acceptor-43565) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=37928,localport=43565], receiveBufferSize:531000
   [junit4]   2> 264195 INFO  (SocketProxy-Acceptor-34761) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42066,localport=33590], receiveBufferSize=530904
   [junit4]   2> 264196 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c  Received 1 versions from http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/ fingerprint:null
   [junit4]   2> 264196 INFO  (SocketProxy-Acceptor-43565) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40267,localport=48978], receiveBufferSize=530904
   [junit4]   2> 264197 INFO  (qtp318993864-2567) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264198 INFO  (qtp2024231405-2588) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&checkCanHandleVersionRanges=false&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 264199 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c  No additional versions requested. ourHighThreshold=1622127286010511360 otherLowThreshold=1622127286010511360 ourHighest=1622127286010511360 otherHighest=1622127286010511360
   [junit4]   2> 264199 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c DONE. sync succeeded
   [junit4]   2> 264199 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 264199 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/: try and ask http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/ to sync
   [junit4]   2> 264202 INFO  (qtp318993864-2563) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264204 INFO  (qtp318993864-2564) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264206 INFO  (qtp2024231405-2591) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t3 url=http://127.0.0.1:34761/k_dpv/c START replicas=[http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/] nUpdates=100
   [junit4]   2> 264208 INFO  (qtp363912069-2540) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 264208 INFO  (qtp363912069-2540) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 264211 INFO  (qtp318993864-2565) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264214 INFO  (qtp2024231405-2591) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 264214 INFO  (qtp2024231405-2591) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 264215 INFO  (qtp2024231405-2591) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&getVersions=100&sync=http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/&wt=javabin&version=2} status=0 QTime=9
   [junit4]   2> 264215 INFO  (qtp318993864-2566) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 264215 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/:  sync completed with http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/
   [junit4]   2> 264216 WARN  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContext The previous leader marked me forceleader_test_collection_shard1_replica_t5 as down and I haven't recovered yet, so I shouldn't be the leader.
   [junit4]   2> 264216 ERROR (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContext There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Leader Initiated Recovery prevented leadership
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.checkLIR(ElectionContext.java:631)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:460)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:57)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:349)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$1(SolrZkClient.java:287)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 264216 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContext There may be a better leader candidate than us - going back into recovery
   [junit4]   2> 264217 INFO  (zkCallback-289-thread-3) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 264218 WARN  (updateExecutor-285-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.RecoveryStrategy Stopping recovery for core=[forceleader_test_collection_shard1_replica_t5] coreNodeName=[core_node6]
   [junit4]   2> 264218 INFO  (updateExecutor-285-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 264218 INFO  (updateExecutor-285-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ActionThrottle Throttling recovery attempts - waiting for 6485ms
   [junit4]   2> 264220 INFO  (qtp318993864-2567) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 264220 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 264220 WARN  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.RecoveryStrategy Stopping recovery for core=[forceleader_test_collection_shard1_replica_t3] coreNodeName=[core_node4]
   [junit4]   2> 264222 INFO  (zkCallback-315-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 264222 INFO  (zkCallback-315-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 264222 INFO  (qtp318993864-2563) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 264222 INFO  (zkCallback-289-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 264222 INFO  (zkCallback-289-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 264222 INFO  (zkCallback-289-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 264222 INFO  (zkCallback-315-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 264224 INFO  (qtp318993864-2564) [n:127.0.0.1:43565_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 264227 INFO  (SocketProxy-Acceptor-44452) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50998,localport=44452], receiveBufferSize:531000
   [junit4]   2> 264229 INFO  (SocketProxy-Acceptor-44452) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33483,localport=50956], receiveBufferSize=530904
   [junit4]   2> 264231 INFO  (qtp363912069-2542) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n1:INDEX.sizeInBytes&key=solr.core.forceleader_test_collection.shard1.replica_t5:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264233 INFO  (qtp363912069-2544) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n1:INDEX.sizeInBytes&key=solr.core.forceleader_test_collection.shard1.replica_t5:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264234 INFO  (qtp363912069-2541) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n1:INDEX.sizeInBytes&key=solr.core.forceleader_test_collection.shard1.replica_t5:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 264237 INFO  (qtp363912069-2540) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264240 INFO  (qtp363912069-2543) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264245 INFO  (qtp363912069-2542) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264247 INFO  (qtp363912069-2544) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264252 INFO  (qtp363912069-2541) [n:127.0.0.1:44452_k_dpv%2Fc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 264267 INFO  (AutoscalingActionExecutor-312-thread-1) [    ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
   [junit4]   2>   "id":"1cd55fe7a15c55T49nze0qkwrapunk67jydvqxoa",
   [junit4]   2>   "source":".auto_add_replicas",
   [junit4]   2>   "eventTime":8115907232619605,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[8115907232619605],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":8115917240305173,
   [junit4]   2>     "nodeNames":["127.0.0.1:37059_k_dpv%2Fc"]}}
   [junit4]   2> 264988 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=0 commError=false errorCode=510 
   [junit4]   2> 264988 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 264988 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 264989 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=1 commError=false errorCode=510 
   [junit4]   2> 264989 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 264989 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 264990 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=2 commError=false errorCode=510 
   [junit4]   2> 264991 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 264991 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 264991 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=3 commError=false errorCode=510 
   [junit4]   2> 264991 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 264991 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 264996 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=4 commError=false errorCode=510 
   [junit4]   2> 264996 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 264996 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 264997 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=5 commError=false errorCode=510 
   [junit4]   2> 264997 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 264997 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase ERROR: org.apache.solr.common.SolrException: Could not find a healthy node to handle the request. ... Sleeping for 1 seconds before re-try ...
   [junit4]   2> 265999 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=0 commError=false errorCode=510 
   [junit4]   2> 265999 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 265999 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 266001 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=1 commError=false errorCode=510 
   [junit4]   2> 266001 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 266001 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 266002 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=2 commError=false errorCode=510 
   [junit4]   2> 266002 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 266002 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 266003 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=3 commError=false errorCode=510 
   [junit4]   2> 266003 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 266004 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 266006 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=4 commError=false errorCode=510 
   [junit4]   2> 266006 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 266007 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 266011 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=5 commError=false errorCode=510 
   [junit4]   2> 266012 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 266012 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase ERROR: org.apache.solr.common.SolrException: Could not find a healthy node to handle the request. ... Sleeping for 1 seconds before re-try ...
   [junit4]   2> 266721 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/
   [junit4]   2> 266721 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t3 url=http://127.0.0.1:34761/k_dpv/c START replicas=[http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/] nUpdates=100
   [junit4]   2> 266723 INFO  (qtp363912069-2540) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 266723 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t3 url=http://127.0.0.1:34761/k_dpv/c  Received 1 versions from http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/ fingerprint:null
   [junit4]   2> 266724 INFO  (qtp363912069-2543) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&checkCanHandleVersionRanges=false&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 266725 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t3 url=http://127.0.0.1:34761/k_dpv/c  No additional versions requested. ourHighThreshold=1622127286010511360 otherLowThreshold=1622127286010511360 ourHighest=1622127286010511360 otherHighest=1622127286010511360
   [junit4]   2> 266725 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t3 url=http://127.0.0.1:34761/k_dpv/c DONE. sync succeeded
   [junit4]   2> 266725 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 266725 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.SyncStrategy http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/: try and ask http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/ to sync
   [junit4]   2> 266728 INFO  (qtp363912069-2542) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c START replicas=[http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/] nUpdates=100
   [junit4]   2> 266731 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.IndexFingerprint IndexFingerprint millis:1.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 266731 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 266732 INFO  (qtp363912069-2542) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 266732 INFO  (qtp363912069-2542) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 266732 INFO  (qtp363912069-2542) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&getVersions=100&sync=http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 266734 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.SyncStrategy http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/:  sync completed with http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/
   [junit4]   2> 266735 WARN  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ShardLeaderElectionContext The previous leader marked me forceleader_test_collection_shard1_replica_t3 as down and I haven't recovered yet, so I shouldn't be the leader.
   [junit4]   2> 266735 ERROR (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ShardLeaderElectionContext There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Leader Initiated Recovery prevented leadership
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.checkLIR(ElectionContext.java:631)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:460)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:57)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:349)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$1(SolrZkClient.java:287)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 266735 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ShardLeaderElectionContext There may be a better leader candidate than us - going back into recovery
   [junit4]   2> 266746 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 266746 WARN  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.RecoveryStrategy Stopping recovery for core=[forceleader_test_collection_shard1_replica_t5] coreNodeName=[core_node6]
   [junit4]   2> 266747 INFO  (zkCallback-315-thread-1) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 266747 WARN  (updateExecutor-311-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.RecoveryStrategy Stopping recovery for core=[forceleader_test_collection_shard1_replica_t3] coreNodeName=[core_node4]
   [junit4]   2> 266752 INFO  (zkCallback-315-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 266752 INFO  (zkCallback-315-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 266752 INFO  (zkCallback-315-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 266752 INFO  (zkCallback-289-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 266752 INFO  (zkCallback-289-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 266752 INFO  (zkCallback-289-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 267014 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=0 commError=false errorCode=510 
   [junit4]   2> 267014 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 267014 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 267016 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=1 commError=false errorCode=510 
   [junit4]   2> 267017 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 267017 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 267019 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=2 commError=false errorCode=510 
   [junit4]   2> 267020 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 267020 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 267022 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=3 commError=false errorCode=510 
   [junit4]   2> 267022 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 267022 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 267024 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=4 commError=false errorCode=510 
   [junit4]   2> 267024 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 267024 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) [forceleader_test_collection] after stale state error from server.
   [junit4]   2> 267026 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection [forceleader_test_collection] failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry=5 commError=false errorCode=510 
   [junit4]   2> 267026 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.CloudSolrClient request was not communication error it seems
   [junit4]   2> 267026 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.AbstractFullDistribZkTestBase No more retries available! Add batch failed due to: org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
   [junit4]   2> 267026 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.ForceLeaderTest Document couldn't be sent, which is expected.
   [junit4]   2> 267038 INFO  (zkConnectionManagerCallback-344-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 267044 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 267048 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39894/solr ready
   [junit4]   2> 267049 INFO  (SocketProxy-Acceptor-34761) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57796,localport=34761], receiveBufferSize:531000
   [junit4]   2> 267051 INFO  (SocketProxy-Acceptor-34761) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42066,localport=33604], receiveBufferSize=530904
   [junit4]   2> 267051 INFO  (qtp2024231405-2590) [n:127.0.0.1:34761_k_dpv%2Fc    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :forceleader with params action=FORCELEADER&collection=forceleader_test_collection&shard=shard1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 267051 INFO  (qtp2024231405-2590) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection   ] o.a.s.h.a.CollectionsHandler Force leader invoked, state: znodeVersion: 0
   [junit4]   2> live nodes:[127.0.0.1:34761_k_dpv%2Fc, 127.0.0.1:43565_k_dpv%2Fc, 127.0.0.1:44452_k_dpv%2Fc]
   [junit4]   2> collections:{collection1=DocCollection(collection1//collections/collection1/state.json/6)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{"core_node4":{
   [junit4]   2>           "core":"collection1_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:43565/k_dpv/c",
   [junit4]   2>           "node_name":"127.0.0.1:43565_k_dpv%2Fc",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true"}}},
   [junit4]   2>     "shard2":{
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"collection1_shard2_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:44452/k_dpv/c",
   [junit4]   2>           "node_name":"127.0.0.1:44452_k_dpv%2Fc",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"true"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "core":"collection1_shard2_replica_n5",
   [junit4]   2>           "base_url":"http://127.0.0.1:34761/k_dpv/c",
   [junit4]   2>           "node_name":"127.0.0.1:34761_k_dpv%2Fc",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT"}}}},
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "tlogReplicas":"0"}, forceleader_test_collection=DocCollection(forceleader_test_collection//collections/forceleader_test_collection/state.json/17)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"0",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t1",
   [junit4]   2>           "base_url":"http://127.0.0.1:37059/k_dpv/c",
   [junit4]   2>           "node_name":"127.0.0.1:37059_k_dpv%2Fc",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"TLOG"},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           "state":"down",
   [junit4]   2>           "base_url":"http://127.0.0.1:34761/k_dpv/c",
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t3",
   [junit4]   2>           "node_name":"127.0.0.1:34761_k_dpv%2Fc",
   [junit4]   2>           "force_set_state":"false",
   [junit4]   2>           "type":"TLOG"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "state":"down",
   [junit4]   2>           "base_url":"http://127.0.0.1:44452/k_dpv/c",
   [junit4]   2>           "core":"forceleader_test_collection_shard1_replica_t5",
   [junit4]   2>           "node_name":"127.0.0.1:44452_k_dpv%2Fc",
   [junit4]   2>           "force_set_state":"false",
   [junit4]   2>           "type":"TLOG"}}}},
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"0",
   [junit4]   2>   "tlogReplicas":"3"}, control_collection=LazyCollectionRef(control_collection)}
   [junit4]   2> 267057 INFO  (qtp2024231405-2590) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection   ] o.a.s.h.a.CollectionsHandler Cleaning out LIR data, which was:     /collections/forceleader_test_collection/leader_initiated_recovery/shard1 (2)
   [junit4]   2>      /collections/forceleader_test_collection/leader_initiated_recovery/shard1/core_node6 (0)
   [junit4]   2>      DATA:
   [junit4]   2>          {
   [junit4]   2>            "state":"down",
   [junit4]   2>            "createdByNodeName":"127.0.0.1:37059_k_dpv%2Fc",
   [junit4]   2>            "createdByCoreNodeName":"core_node2"}
   [junit4]   2>      /collections/forceleader_test_collection/leader_initiated_recovery/shard1/core_node4 (0)
   [junit4]   2>      DATA:
   [junit4]   2>          {
   [junit4]   2>            "state":"down",
   [junit4]   2>            "createdByNodeName":"127.0.0.1:37059_k_dpv%2Fc",
   [junit4]   2>            "createdByCoreNodeName":"core_node2"}
   [junit4]   2> 
   [junit4]   2> 268782 INFO  (recoveryExecutor-312-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=false
   [junit4]   2> 268783 INFO  (recoveryExecutor-312-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.ZkController forceleader_test_collection_shard1_replica_t3 stopping background replication from leader
   [junit4]   2> 269246 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/
   [junit4]   2> 269247 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c START replicas=[http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/] nUpdates=100
   [junit4]   2> 269248 INFO  (qtp2024231405-2588) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 269249 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c  Received 1 versions from http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/ fingerprint:null
   [junit4]   2> 269249 INFO  (qtp2024231405-2591) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&checkCanHandleVersionRanges=false&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 269250 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c  No additional versions requested. ourHighThreshold=1622127286010511360 otherLowThreshold=1622127286010511360 ourHighest=1622127286010511360 otherHighest=1622127286010511360
   [junit4]   2> 269250 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t5 url=http://127.0.0.1:44452/k_dpv/c DONE. sync succeeded
   [junit4]   2> 269250 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 269250 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/: try and ask http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/ to sync
   [junit4]   2> 269252 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync PeerSync: core=forceleader_test_collection_shard1_replica_t3 url=http://127.0.0.1:34761/k_dpv/c START replicas=[http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/] nUpdates=100
   [junit4]   2> 269253 INFO  (qtp363912069-2544) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 269253 INFO  (qtp363912069-2544) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 269254 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 269254 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 269254 INFO  (qtp2024231405-2589) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t3]  webapp=/k_dpv/c path=/get params={distrib=false&qt=/get&getVersions=100&sync=http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 269255 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SyncStrategy http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/:  sync completed with http://127.0.0.1:34761/k_dpv/c/forceleader_test_collection_shard1_replica_t3/
   [junit4]   2> 269255 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ZkController forceleader_test_collection_shard1_replica_t5 stopping background replication from leader
   [junit4]   2> 269256 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContext Replaying tlog before become new leader
   [junit4]   2> 269256 WARN  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.UpdateLog Starting log replay tlog{file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-repro/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_1B45B21C0A901F9-001/shard-1-001/cores/forceleader_test_collection_shard1_replica_t5/data/tlog/tlog.0000000000000000000 refcount=2} active=false starting pos=0 inSortedOrder=true
   [junit4]   2> 269261 INFO  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.DirectUpdateHandler2 start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 269261 INFO  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@47ef304 commitCommandVersion:0
   [junit4]   2> 269291 INFO  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.s.SolrIndexSearcher Opening [Searcher@2122dfb5[forceleader_test_collection_shard1_replica_t5] main]
   [junit4]   2> 269294 INFO  (searcherExecutor-301-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.SolrCore [forceleader_test_collection_shard1_replica_t5] Registered new searcher Searcher@2122dfb5[forceleader_test_collection_shard1_replica_t5] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.7.0):c1:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546981148132}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
   [junit4]   2> 269296 INFO  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 269296 INFO  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.p.LogUpdateProcessorFactory [forceleader_test_collection_shard1_replica_t5] {add=[1 (1622127286010511360)]} 0 40
   [junit4]   2> 269296 WARN  (recoveryExecutor-303-thread-1-processing-n:127.0.0.1:44452_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t5 c:forceleader_test_collection s:shard1 r:core_node6) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=1 deletes=0 deleteByQuery=0 errors=0 positionOfStart=0}
   [junit4]   2> 269296 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/forceleader_test_collection/leaders/shard1/leader after winning as /collections/forceleader_test_collection/leader_elect/shard1/election/72589476886609930-core_node6-n_0000000006
   [junit4]   2> 269303 INFO  (zkCallback-289-thread-1) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44452/k_dpv/c/forceleader_test_collection_shard1_replica_t5/ shard1
   [junit4]   2> 269303 INFO  (zkCallback-315-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 269303 INFO  (zkCallback-289-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 269303 INFO  (zkCallback-315-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 269303 INFO  (zkCallback-315-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 269303 INFO  (zkCallback-289-thread-4) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 269303 INFO  (zkCallback-289-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forceleader_test_collection/state.json] for collection [forceleader_test_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 269305 INFO  (SocketProxy-Acceptor-44452) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51008,localport=44452], receiveBufferSize:531000
   [junit4]   2> 269307 INFO  (SocketProxy-Acceptor-44452) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33483,localport=50966], receiveBufferSize=530904
   [junit4]   2> 269326 INFO  (qtp363912069-2540) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/admin/ping params={wt=javabin&version=2} hits=1 status=0 QTime=19
   [junit4]   2> 269326 INFO  (qtp363912069-2540) [n:127.0.0.1:44452_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node6 x:forceleader_test_collection_shard1_replica_t5] o.a.s.c.S.Request [forceleader_test_collection_shard1_replica_t5]  webapp=/k_dpv/c path=/admin/ping params={wt=javabin&version=2} status=0 QTime=19
   [junit4]   2> 269327 INFO  (recoveryExecutor-312-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[forceleader_test_collection_shard1_replica_t3]
   [junit4]   2> 269328 INFO  (recoveryExecutor-312-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-repro/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_1B45B21C0A901F9-001/shard-3-001/cores/forceleader_test_collection_shard1_replica_t3/data/tlog/tlog.0000000000000000000 refcount=1}}
   [junit4]   2> 269328 INFO  (recoveryExecutor-312-thread-1-processing-n:127.0.0.1:34761_k_dpv%2Fc x:forceleader_test_collection_shard1_replica_t3 c:forceleader_test_collection s:shard1 r:core_node4) [n:127.0.0.1:34761_k_dpv%2Fc c:forceleader_test_collection s:shard1 r:core_node4 x:forceleader_test_collection_shard1_replica_t3] o.a.s.c.RecoveryStrategy Publishing state of core [forceleader_test_collection_shard1_replica_t3] as recovering,

[...truncated too long message...]

   [junit4]   2> 1100777 INFO  (zkCallback-702-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1100781 INFO  (closeThreadPool-743-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1100781 INFO  (closeThreadPool-743-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5c24e471: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@326dfcfb
   [junit4]   2> 1100781 INFO  (closeThreadPool-743-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1100781 INFO  (closeThreadPool-743-thread-7) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1100781 INFO  (closeThreadPool-743-thread-7) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4219f84b: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@326dfcfb
   [junit4]   2> 1100781 INFO  (closeThreadPool-743-thread-7) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1100784 INFO  (closeThreadPool-743-thread-6) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6535d5bd{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1100785 INFO  (closeThreadPool-743-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5586889{/k_dpv/c,null,UNAVAILABLE}
   [junit4]   2> 1100788 INFO  (zkCallback-709-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1100797 INFO  (closeThreadPool-743-thread-6) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 1100798 INFO  (zkCallback-702-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1100798 INFO  (zkCallback-737-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1100805 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x101e3cb388b0018, likely client has closed socket
   [junit4]   2> 1100806 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x101e3cb388b0012, likely client has closed socket
   [junit4]   2> 1100806 INFO  (zkCallback-709-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 1100808 WARN  (closeThreadPool-743-thread-6) [    ] o.a.s.c.s.c.SocketProxy Closing 8 connections to: http://127.0.0.1:35943/k_dpv/c, target: http://127.0.0.1:35244/k_dpv/c
   [junit4]   2> 1100811 INFO  (closeThreadPool-743-thread-7) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e828a3d{HTTP/1.1,[http/1.1]}{127.0.0.1:44512}
   [junit4]   2> 1100813 INFO  (closeThreadPool-743-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7b106764{/k_dpv/c,null,UNAVAILABLE}
   [junit4]   2> 1100813 INFO  (closeThreadPool-743-thread-7) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 1100815 INFO  (closeThreadPool-743-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@61579449{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1100815 WARN  (closeThreadPool-743-thread-7) [    ] o.a.s.c.s.c.SocketProxy Closing 4 connections to: http://127.0.0.1:46469/k_dpv/c, target: http://127.0.0.1:44512/k_dpv/c
   [junit4]   2> 1100815 INFO  (closeThreadPool-743-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@481bbc17{/k_dpv/c,null,UNAVAILABLE}
   [junit4]   2> 1100816 INFO  (closeThreadPool-743-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 1100817 WARN  (closeThreadPool-743-thread-1) [    ] o.a.s.c.s.c.SocketProxy Closing 12 connections to: http://127.0.0.1:37353/k_dpv/c, target: http://127.0.0.1:43605/k_dpv/c
   [junit4]   2> 1100817 INFO  (TEST-ForceLeaderTest.testReplicasInLowerTerms-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 1100825 WARN  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	44	/solr/collections/forceleader_lower_terms_collection/terms/shard1
   [junit4]   2> 	21	/solr/collections/forceleader_lower_terms_collection/state.json
   [junit4]   2> 	17	/solr/collections/collection1/state.json
   [junit4]   2> 	14	/solr/collections/collection1/terms/shard2
   [junit4]   2> 	13	/solr/aliases.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/configs/conf1
   [junit4]   2> 	2	/solr/collections/collection1/terms/shard1
   [junit4]   2> 	2	/solr/collections/control_collection/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	13	/solr/clusterprops.json
   [junit4]   2> 	13	/solr/clusterstate.json
   [junit4]   2> 	6	/solr/collections/control_collection/state.json
   [junit4]   2> 	2	/solr/collections/collection1/leader_elect/shard2/election/72589530981138450-core_node4-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/72589530981138443-127.0.0.1:35943_k_dpv%2Fc-n_0000000001
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	13	/solr/collections
   [junit4]   2> 	12	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 1100836 INFO  (TEST-ForceLeaderTest.testReplicasInLowerTerms-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45107
   [junit4]   2> 1100836 INFO  (TEST-ForceLeaderTest.testReplicasInLowerTerms-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 45107
   [junit4] OK      40.3s J2 | ForceLeaderTest.testReplicasInLowerTerms
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-repro/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_1B45B21C0A901F9-002
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {range_facet_l_dv=Lucene50(blocksize=128), multiDefault=BlockTreeOrds(blocksize=128), a_t=BlockTreeOrds(blocksize=128), intDefault=PostingsFormat(name=Memory), id_i1=PostingsFormat(name=Memory), id=Lucene50(blocksize=128), range_facet_i_dv=BlockTreeOrds(blocksize=128), text=PostingsFormat(name=Direct), range_facet_l=BlockTreeOrds(blocksize=128), timestamp=BlockTreeOrds(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Direct), multiDefault=DocValuesFormat(name=Lucene70), a_t=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Direct), id_i1=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), text=DocValuesFormat(name=Memory), intDvoDefault=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70), range_facet_l=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1146, maxMBSortInHeap=5.8424922619614446, sim=RandomSimilarity(queryNorm=true): {}, locale=es-PA, timezone=US/Arizona
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=265343560,total=503840768
   [junit4]   2> NOTE: All tests run in this JVM: [ForceLeaderTest, ForceLeaderTest, TestDistributedSearch, ForceLeaderTest]
   [junit4] Completed [9/10 (4!)] on J2 in 195.02s, 3 tests, 1 error, 1 skipped <<< FAILURES!
   [junit4] 
   [junit4] Suite: org.apache.solr.TestDistributedSearch
   [junit4] OK       290s J1 | TestDistributedSearch.test
   [junit4] Completed [10/10 (4!)] on J1 in 290.28s, 1 test
   [junit4] 
   [junit4] 
   [junit4] Tests with failures [seed: 1B45B21C0A901F9]:
   [junit4]   - org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader
   [junit4]   - org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader
   [junit4]   - org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader
   [junit4]   - org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     1.05 ..  1097.21 =  1096.16s
   [junit4] JVM J1:     0.89 ..  1177.53 =  1176.64s
   [junit4] JVM J2:     0.89 ..  1154.25 =  1153.36s
   [junit4] Execution time total: 19 minutes 37 seconds
   [junit4] Tests summary: 10 suites, 20 tests, 4 errors, 5 ignored

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-repro/lucene/common-build.xml:1572: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-repro/lucene/common-build.xml:1099: There were test failures: 10 suites, 20 tests, 4 errors, 5 ignored [seed: 1B45B21C0A901F9]

Total time: 19 minutes 40 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.TestDistributedSearch
[repro]   4/5 failed: org.apache.solr.cloud.ForceLeaderTest
[repro] git checkout 951b4e4c83756d2d5b8592168cdba828a9133ba3
Previous HEAD position was 5c813f3... SOLR-7896: Avoid browser basicAuth dialogue when blockUnknown=false. Always show Dashboard menu. Clarify refGuide
HEAD is now at 951b4e4... SOLR-13053: Upgrade CHANGES.txt
[repro] Exiting with code 256
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-repro Repro-Lucene-Solr-Tests-master#3130 to compare, so performing full copy of artifacts
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

Mime
View raw message