lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1030 - Still Failing
Date Thu, 02 Jun 2016 05:47:57 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1030/

8 tests failed.
FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testRandomUpdates

Error Message:
org.apache.solr.client.solrj.impl.HttpSolrClient@67604f9 => [type=<DELID>,id=<id_50179>,msgSubStr=<null>, type=<ADD>,id=<id_49929>,msgSubStr=<null>, type=<DELID>,id=<id_15498>,msgSubStr=<null>]: Num error ids: [{type=DELID,id=id_50179,message=version conflict for id_50179 expected=1 actual=-1}, {type=DELID,id=id_15498,message=version conflict for id_15498 expected=1 actual=-1}] expected:<3> but was:<2>

Stack Trace:
java.lang.AssertionError: org.apache.solr.client.solrj.impl.HttpSolrClient@67604f9 => [type=<DELID>,id=<id_50179>,msgSubStr=<null>, type=<ADD>,id=<id_49929>,msgSubStr=<null>, type=<DELID>,id=<id_15498>,msgSubStr=<null>]: Num error ids: [{type=DELID,id=id_50179,message=version conflict for id_50179 expected=1 actual=-1}, {type=DELID,id=id_15498,message=version conflict for id_15498 expected=1 actual=-1}] expected:<3> but was:<2>
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB:69140BE227ABBDD9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertUpdateTolerantErrors(TestTolerantUpdateProcessorCloud.java:974)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testRandomUpdates(TestTolerantUpdateProcessorRandomCloud.java:266)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testSanityRandomUnsetBit

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB:D676661A11C2AD31]:0)
	at sun.security.ssl.InputRecord.<init>(InputRecord.java:93)
	at sun.security.ssl.AppInputStream.<init>(AppInputStream.java:50)
	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:635)
	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1228)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:998)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:934)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.deleteAllDocs(TestTolerantUpdateProcessorRandomCloud.java:130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud:     1) Thread[id=11670, name=OverseerHdfsCoreFailoverThread-96000019652608024-127.0.0.1:56818_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud: 
   1) Thread[id=11670, name=OverseerHdfsCoreFailoverThread-96000019652608024-127.0.0.1:56818_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=11670, name=OverseerHdfsCoreFailoverThread-96000019652608024-127.0.0.1:56818_solr-n_0000000001, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=11670, name=OverseerHdfsCoreFailoverThread-96000019652608024-127.0.0.1:56818_solr-n_0000000001, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:258)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=150492, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=153064, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=150147, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=155515, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=149363, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=155021, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=150490, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=155386, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=147967, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=155757, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=149362, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=150146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=155514, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=147969, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=150914, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=149815, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=153065, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=155760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=149364, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=155517, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=155383, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=155385, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=153067, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=155758, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=149816, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=153066, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=155759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=147968, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=155019, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=152407, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=152408, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=150913, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=149813, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=147970, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=149814, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=149361, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=150489, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=150915, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=150148, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=155022, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=155384, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=155516, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=152409, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=155020, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=150491, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=150149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=150912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=152410, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=150492, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=153064, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=150147, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=155515, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=149363, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=155021, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=150490, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=155386, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=147967, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=155757, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=149362, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=150146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=155514, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=147969, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=150914, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=149815, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=153065, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=155760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=149364, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=155517, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=155383, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=155385, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=153067, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=155758, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=149816, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=153066, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=155759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=147968, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=155019, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=152407, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=152408, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=150913, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=149813, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=147970, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=149814, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=149361, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=150489, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=150915, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=150148, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=155022, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=155384, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=155516, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=152409, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=155020, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=150491, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=150149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=150912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=152410, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:258)
	at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=108190, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=108191, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=108629, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=108626, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=108192, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=108627, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=108193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=108628, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=108190, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=108191, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=108629, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=108626, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=108192, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=108627, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=108193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=108628, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)




Build Log:
[...truncated 11218 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/init-core-data-001
   [junit4]   2> 308896 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 308900 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.TestTolerantUpdateProcessorRandomCloud Configuring cluster: servers=13, shards=4, repfactor=3
   [junit4]   2> 308901 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 308903 INFO  (Thread-9920) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 308905 INFO  (Thread-9920) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 309003 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44345
   [junit4]   2> 309004 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309006 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309055 INFO  (zkCallback-118-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@43d22125 name:ZooKeeperConnection Watcher:127.0.0.1:44345 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309055 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309056 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309056 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 309057 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[55D5DE5250D643CB]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/clusterprops.json
   [junit4]   2> 309149 INFO  (jetty-launcher-117-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309149 INFO  (jetty-launcher-117-thread-7) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309149 INFO  (jetty-launcher-117-thread-3) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309150 INFO  (jetty-launcher-117-thread-4) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309150 INFO  (jetty-launcher-117-thread-5) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309150 INFO  (jetty-launcher-117-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309151 INFO  (jetty-launcher-117-thread-12) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309152 INFO  (jetty-launcher-117-thread-11) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309152 INFO  (jetty-launcher-117-thread-10) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309153 INFO  (jetty-launcher-117-thread-9) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309154 INFO  (jetty-launcher-117-thread-8) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309206 INFO  (jetty-launcher-117-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@75af7797{/solr,null,AVAILABLE}
   [junit4]   2> 309338 INFO  (jetty-launcher-117-thread-13) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309367 INFO  (jetty-launcher-117-thread-8) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2270ee26{/solr,null,AVAILABLE}
   [junit4]   2> 309387 INFO  (jetty-launcher-117-thread-9) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d7da02{/solr,null,AVAILABLE}
   [junit4]   2> 309405 INFO  (jetty-launcher-117-thread-6) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 309407 INFO  (jetty-launcher-117-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2a12c97c{/solr,null,AVAILABLE}
   [junit4]   2> 309408 INFO  (jetty-launcher-117-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3fbea40e{/solr,null,AVAILABLE}
   [junit4]   2> 309444 INFO  (jetty-launcher-117-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@689b397a{/solr,null,AVAILABLE}
   [junit4]   2> 309448 INFO  (jetty-launcher-117-thread-13) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b280b03{/solr,null,AVAILABLE}
   [junit4]   2> 309452 INFO  (jetty-launcher-117-thread-12) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2add5ad{/solr,null,AVAILABLE}
   [junit4]   2> 309454 INFO  (jetty-launcher-117-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@46ad3216{/solr,null,AVAILABLE}
   [junit4]   2> 309455 INFO  (jetty-launcher-117-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a925880{/solr,null,AVAILABLE}
   [junit4]   2> 309455 INFO  (jetty-launcher-117-thread-10) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62e51882{/solr,null,AVAILABLE}
   [junit4]   2> 309460 INFO  (jetty-launcher-117-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@352c5075{/solr,null,AVAILABLE}
   [junit4]   2> 309461 INFO  (jetty-launcher-117-thread-11) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@275f74d7{/solr,null,AVAILABLE}
   [junit4]   2> 309562 INFO  (jetty-launcher-117-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@29dec4d6{SSL,[ssl, http/1.1]}{127.0.0.1:48939}
   [junit4]   2> 309562 INFO  (jetty-launcher-117-thread-1) [    ] o.e.j.s.Server Started @312703ms
   [junit4]   2> 309562 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48939}
   [junit4]   2> 309563 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309563 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node1'
   [junit4]   2> 309563 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309563 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309577 INFO  (jetty-launcher-117-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@3eb71d4b{SSL,[ssl, http/1.1]}{127.0.0.1:57516}
   [junit4]   2> 309577 INFO  (jetty-launcher-117-thread-4) [    ] o.e.j.s.Server Started @312718ms
   [junit4]   2> 309577 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57516}
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node4'
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.e.j.s.ServerConnector Started ServerConnector@720be64a{SSL,[ssl, http/1.1]}{127.0.0.1:58817}
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.e.j.s.Server Started @312719ms
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58817}
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node10'
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309578 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309579 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309579 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309579 INFO  (jetty-launcher-117-thread-9) [    ] o.e.j.s.ServerConnector Started ServerConnector@9c3897{SSL,[ssl, http/1.1]}{127.0.0.1:43621}
   [junit4]   2> 309579 INFO  (jetty-launcher-117-thread-9) [    ] o.e.j.s.Server Started @312720ms
   [junit4]   2> 309579 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43621}
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node9'
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309580 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309607 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309616 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309617 INFO  (zkCallback-119-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@64115997 name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309618 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309618 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309618 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309621 INFO  (jetty-launcher-117-thread-6) [    ] o.e.j.s.ServerConnector Started ServerConnector@33619200{SSL,[ssl, http/1.1]}{127.0.0.1:50692}
   [junit4]   2> 309621 INFO  (jetty-launcher-117-thread-6) [    ] o.e.j.s.Server Started @312762ms
   [junit4]   2> 309621 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50692}
   [junit4]   2> 309621 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309622 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node6'
   [junit4]   2> 309622 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309622 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309622 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309623 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309631 INFO  (zkCallback-122-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@199a30f5 name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309631 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309632 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309632 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@6dc7bc7e{SSL,[ssl, http/1.1]}{127.0.0.1:56818}
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.e.j.s.Server Started @312787ms
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56818}
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node2'
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309646 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309647 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309647 INFO  (zkCallback-120-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4bf7bbfc name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309647 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309647 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309647 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309656 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309656 INFO  (jetty-launcher-117-thread-12) [    ] o.e.j.s.ServerConnector Started ServerConnector@151120e3{SSL,[ssl, http/1.1]}{127.0.0.1:59071}
   [junit4]   2> 309656 INFO  (jetty-launcher-117-thread-12) [    ] o.e.j.s.Server Started @312797ms
   [junit4]   2> 309656 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59071}
   [junit4]   2> 309656 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309657 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node12'
   [junit4]   2> 309657 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309657 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309657 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309659 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309659 INFO  (jetty-launcher-117-thread-5) [    ] o.e.j.s.ServerConnector Started ServerConnector@6137008a{SSL,[ssl, http/1.1]}{127.0.0.1:44517}
   [junit4]   2> 309659 INFO  (jetty-launcher-117-thread-5) [    ] o.e.j.s.Server Started @312800ms
   [junit4]   2> 309659 INFO  (jetty-launcher-117-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44517}
   [junit4]   2> 309660 INFO  (jetty-launcher-117-thread-5) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309660 INFO  (jetty-launcher-117-thread-5) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node5'
   [junit4]   2> 309660 INFO  (jetty-launcher-117-thread-5) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309660 INFO  (jetty-launcher-117-thread-5) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309660 INFO  (jetty-launcher-117-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309667 INFO  (jetty-launcher-117-thread-11) [    ] o.e.j.s.ServerConnector Started ServerConnector@3ab98e28{SSL,[ssl, http/1.1]}{127.0.0.1:41373}
   [junit4]   2> 309667 INFO  (jetty-launcher-117-thread-11) [    ] o.e.j.s.Server Started @312808ms
   [junit4]   2> 309667 INFO  (zkCallback-121-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d5b72ad name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309667 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41373}
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node11'
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309668 INFO  (jetty-launcher-117-thread-9) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309671 INFO  (zkCallback-124-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@30df16ca name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309671 INFO  (jetty-launcher-117-thread-7) [    ] o.e.j.s.ServerConnector Started ServerConnector@1b01c6f4{SSL,[ssl, http/1.1]}{127.0.0.1:44840}
   [junit4]   2> 309671 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309671 INFO  (jetty-launcher-117-thread-7) [    ] o.e.j.s.Server Started @312812ms
   [junit4]   2> 309671 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44840}
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node7'
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309672 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309683 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309683 INFO  (jetty-launcher-117-thread-13) [    ] o.e.j.s.ServerConnector Started ServerConnector@66e94094{SSL,[ssl, http/1.1]}{127.0.0.1:59399}
   [junit4]   2> 309683 INFO  (jetty-launcher-117-thread-8) [    ] o.e.j.s.ServerConnector Started ServerConnector@21a5b78d{SSL,[ssl, http/1.1]}{127.0.0.1:46427}
   [junit4]   2> 309683 INFO  (jetty-launcher-117-thread-13) [    ] o.e.j.s.Server Started @312824ms
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.e.j.s.Server Started @312824ms
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59399}
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46427}
   [junit4]   2> 309683 INFO  (jetty-launcher-117-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@51f38f4a{SSL,[ssl, http/1.1]}{127.0.0.1:37952}
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.e.j.s.Server Started @312825ms
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37952}
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node13'
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node8'
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node3'
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-8) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309684 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 309794 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309795 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node4/.
   [junit4]   2> 309795 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.CoreContainer New CoreContainer 612514175
   [junit4]   2> 309795 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node4]
   [junit4]   2> 309795 WARN  (jetty-launcher-117-thread-4) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node4/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node4/lib
   [junit4]   2> 309815 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node10/.
   [junit4]   2> 309815 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.CoreContainer New CoreContainer 1306078236
   [junit4]   2> 309815 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node10]
   [junit4]   2> 309815 WARN  (jetty-launcher-117-thread-10) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node10/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node10/lib
   [junit4]   2> 309816 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 309816 INFO  (jetty-launcher-117-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 309815 INFO  (zkCallback-125-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3b5f8ef6 name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309814 INFO  (zkCallback-123-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2f465c98 name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309835 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309836 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309795 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node1/.
   [junit4]   2> 309836 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 923307894
   [junit4]   2> 309836 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node1]
   [junit4]   2> 309836 WARN  (jetty-launcher-117-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node1/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_55D5DE5250D643CB-001/tempDir-001/node1/lib
   [junit4]   2> 309836 INFO  (jetty-launcher-117-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 309836 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309836 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309837 INFO  (jetty-launcher-117-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309839 INFO  (jetty-launcher-117-thread-10) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 309841 INFO  (jetty-launcher-117-thread-12) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309848 INFO  (zkCallback-128-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5c0f003d name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309848 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309848 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309848 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309851 INFO  (jetty-launcher-117-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309853 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309853 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d8b759c name:ZooKeeperConnection Watcher:127.0.0.1:44345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309853 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 309853 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 309853 INFO  (jetty-launcher-117-thread-13) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309854 INFO  (jetty-launcher-117-thread-11) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 309854 INFO  (jetty-launcher-117-thread-7) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directo

[...truncated too long message...]

 java.lang.Thread.run(Thread.java:745)
   [junit4]    >   27) Thread[id=155759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   28) Thread[id=147968, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   29) Thread[id=155019, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   30) Thread[id=152407, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   31) Thread[id=152408, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   32) Thread[id=150913, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   33) Thread[id=149813, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   34) Thread[id=147970, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   35) Thread[id=149814, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   36) Thread[id=149361, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   37) Thread[id=150489, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   38) Thread[id=150915, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   39) Thread[id=150148, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   40) Thread[id=155022, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   41) Thread[id=155384, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   42) Thread[id=155516, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   43) Thread[id=152409, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   44) Thread[id=155020, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   45) Thread[id=150491, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   46) Thread[id=150149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   47) Thread[id=150912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   48) Thread[id=152410, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([55D5DE5250D643CB]:0)
   [junit4] Completed [544/611 (3!)] on J0 in 1029.85s, 11 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 221 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:531: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1427: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: There were test failures: 611 suites (10 ignored), 2559 tests, 6 suite-level errors, 1 error, 1 failure, 1665 ignored (8 assumptions) [seed: 55D5DE5250D643CB]

Total time: 202 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




Mime
View raw message