lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build # 10559 - Still Failing!
Date Sat, 14 Jun 2014 19:46:15 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10559/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:451)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
	at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest:     1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    2) Thread[id=6672, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-EventThread, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   2) Thread[id=6672, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-EventThread, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9]:0)


REGRESSION:  org.apache.solr.cloud.DeleteInactiveReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node2 in shard/collection : shard1/delDeadColl available replicas are core_node1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node2 in shard/collection : shard1/delDeadColl available replicas are core_node1
	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9:95ABC59FFFEC495]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:659)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:122)
	at org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest(DeleteInactiveReplicaTest.java:127)
	at org.apache.solr.cloud.DeleteInactiveReplicaTest.doTest(DeleteInactiveReplicaTest.java:46)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11794 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteInactiveReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/init-core-data-001
   [junit4]   2> 1472673 T6587 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1472674 T6587 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /wllr/l
   [junit4]   2> 1472676 T6587 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1472677 T6587 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1472677 T6588 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1472777 T6587 oasc.ZkTestServer.run start zk server on port:39032
   [junit4]   2> 1472778 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1481479 T6591 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 8700ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 1481481 T6594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25befa17 name:ZooKeeperConnection Watcher:127.0.0.1:39032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1481482 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1481482 T6587 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1481487 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1481488 T6596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@550edc84 name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1481489 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1481489 T6587 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1481493 T6587 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1481495 T6587 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1481497 T6587 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1481499 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1481499 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1481502 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1481503 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1481505 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1481505 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1481506 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1481507 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1481508 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1481509 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1481510 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1481510 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1481511 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1481512 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1481513 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1481513 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1481514 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1481515 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1481516 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1481516 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1481517 T6587 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1481518 T6587 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1481747 T6587 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1481751 T6587 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38248
   [junit4]   2> 1481751 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1481752 T6587 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1481752 T6587 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002
   [junit4]   2> 1481752 T6587 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/'
   [junit4]   2> 1481762 T6587 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/solr.xml
   [junit4]   2> 1481776 T6587 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002
   [junit4]   2> 1481776 T6587 oasc.CoreContainer.<init> New CoreContainer 1499713645
   [junit4]   2> 1481777 T6587 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/]
   [junit4]   2> 1481777 T6587 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1481777 T6587 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1481778 T6587 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1481778 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1481778 T6587 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1481778 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1481778 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1481779 T6587 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1481779 T6587 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1481779 T6587 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1481780 T6587 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1481780 T6587 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1481780 T6587 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39032/solr
   [junit4]   2> 1481780 T6587 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1481781 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1481782 T6607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3067fb5f name:ZooKeeperConnection Watcher:127.0.0.1:39032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1481782 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1481784 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1481785 T6609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a36ebb5 name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1481785 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1481786 T6587 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1481788 T6587 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1481790 T6587 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1481791 T6587 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1481793 T6587 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1481794 T6587 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1481795 T6587 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38248_wllr%2Fl
   [junit4]   2> 1481796 T6587 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38248_wllr%2Fl
   [junit4]   2> 1481802 T6587 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1481804 T6587 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1481807 T6587 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38248_wllr%2Fl
   [junit4]   2> 1481807 T6587 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1481810 T6587 oasc.Overseer.start Overseer (id=91932264779481091-127.0.0.1:38248_wllr%2Fl-n_0000000000) starting
   [junit4]   2> 1481813 T6587 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1481818 T6611 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1481818 T6587 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1481820 T6587 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1481820 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1481822 T6610 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1481823 T6587 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002
   [junit4]   2> 1481829 T6587 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/conf/
   [junit4]   2> 1481951 T6587 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1481952 T6587 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1481952 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1481953 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1481968 T6614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ab8023f name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1481968 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1481969 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1481970 T6587 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1482219 T6587 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1482219 T6587 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1482221 T6587 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41592
   [junit4]   2> 1482221 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1482222 T6587 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1482222 T6587 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003
   [junit4]   2> 1482222 T6587 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003/'
   [junit4]   2> 1482234 T6587 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003/solr.xml
   [junit4]   2> 1482248 T6587 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003
   [junit4]   2> 1482249 T6587 oasc.CoreContainer.<init> New CoreContainer 1637371423
   [junit4]   2> 1482249 T6587 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003/]
   [junit4]   2> 1482249 T6587 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1482250 T6587 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1482250 T6587 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1482250 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1482250 T6587 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1482251 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1482251 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1482251 T6587 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1482251 T6587 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1482252 T6587 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1482252 T6587 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1482252 T6587 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1482253 T6587 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39032/solr
   [junit4]   2> 1482253 T6587 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1482254 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1482255 T6625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2069e6bb name:ZooKeeperConnection Watcher:127.0.0.1:39032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1482255 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1482257 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1482257 T6627 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f5d10a8 name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1482258 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1482261 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1483263 T6587 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41592_wllr%2Fl
   [junit4]   2> 1483264 T6587 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41592_wllr%2Fl
   [junit4]   2> 1483268 T6587 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003
   [junit4]   2> 1483276 T6587 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003/conf/
   [junit4]   2> 1483400 T6587 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1483416 T6587 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1483416 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1483730 T6587 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1483731 T6587 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1483736 T6587 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54343
   [junit4]   2> 1483737 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1483738 T6587 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1483739 T6587 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004
   [junit4]   2> 1483739 T6587 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004/'
   [junit4]   2> 1483759 T6587 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004/solr.xml
   [junit4]   2> 1483780 T6587 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004
   [junit4]   2> 1483780 T6587 oasc.CoreContainer.<init> New CoreContainer 1454998011
   [junit4]   2> 1483781 T6587 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004/]
   [junit4]   2> 1483781 T6587 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1483782 T6587 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1483782 T6587 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1483783 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1483783 T6587 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1483783 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1483784 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1483784 T6587 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1483784 T6587 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1483785 T6587 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1483786 T6587 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1483786 T6587 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1483786 T6587 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39032/solr
   [junit4]   2> 1483787 T6587 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1483788 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1483790 T6639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60b815ce name:ZooKeeperConnection Watcher:127.0.0.1:39032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1483790 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1483801 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1483801 T6641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@466ce104 name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1483802 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1483812 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1484815 T6587 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54343_wllr%2Fl
   [junit4]   2> 1484816 T6587 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54343_wllr%2Fl
   [junit4]   2> 1484820 T6587 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004
   [junit4]   2> 1484824 T6587 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-004/conf/
   [junit4]   2> 1484918 T6587 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1484919 T6587 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1484919 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1485158 T6587 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1485159 T6587 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1485161 T6587 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40787
   [junit4]   2> 1485161 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1485161 T6587 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1485162 T6587 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005
   [junit4]   2> 1485162 T6587 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/'
   [junit4]   2> 1485172 T6587 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/solr.xml
   [junit4]   2> 1485187 T6587 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005
   [junit4]   2> 1485187 T6587 oasc.CoreContainer.<init> New CoreContainer 375281480
   [junit4]   2> 1485187 T6587 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/]
   [junit4]   2> 1485188 T6587 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1485188 T6587 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1485188 T6587 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1485188 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1485189 T6587 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1485189 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1485189 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1485189 T6587 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1485190 T6587 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1485190 T6587 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1485190 T6587 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1485191 T6587 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1485191 T6587 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39032/solr
   [junit4]   2> 1485191 T6587 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1485192 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1485193 T6653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ca3156a name:ZooKeeperConnection Watcher:127.0.0.1:39032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1485193 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1485194 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1485195 T6655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2be97536 name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1485195 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1485199 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1486200 T6587 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40787_wllr%2Fl
   [junit4]   2> 1486202 T6587 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40787_wllr%2Fl
   [junit4]   2> 1486205 T6587 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005
   [junit4]   2> 1486210 T6587 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/conf/
   [junit4]   2> 1486321 T6587 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1486322 T6587 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1486322 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1486557 T6587 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1486558 T6587 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1486560 T6587 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41494
   [junit4]   2> 1486560 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1486561 T6587 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1486561 T6587 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006
   [junit4]   2> 1486561 T6587 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/'
   [junit4]   2> 1486580 T6587 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/solr.xml
   [junit4]   2> 1486589 T6587 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006
   [junit4]   2> 1486590 T6587 oasc.CoreContainer.<init> New CoreContainer 1217914987
   [junit4]   2> 1486590 T6587 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/]
   [junit4]   2> 1486591 T6587 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1486591 T6587 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1486591 T6587 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1486591 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1486592 T6587 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1486592 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1486592 T6587 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1486592 T6587 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1486593 T6587 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1486593 T6587 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1486593 T6587 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1486594 T6587 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1486594 T6587 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39032/solr
   [junit4]   2> 1486595 T6587 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1486596 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1486596 T6667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65c86df0 name:ZooKeeperConnection Watcher:127.0.0.1:39032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1486596 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1486598 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1486598 T6669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c132a39 name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1486599 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1486609 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1487611 T6587 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41494_wllr%2Fl
   [junit4]   2> 1487612 T6587 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41494_wllr%2Fl
   [junit4]   2> 1487617 T6587 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006
   [junit4]   2> 1487623 T6587 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/conf/
   [junit4]   2> 1487744 T6587 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1487745 T6587 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1487745 T6587 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1487746 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1487747 T6672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b6cfe6c name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1487747 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1487748 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1487752 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487752 T6605 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=clusterprop&wt=javabin&version=2} status=0 QTime=1 
   [junit4]   2> 1487753 T6610 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1487755 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487755 T6627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487755 T6672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487755 T6669 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487755 T6614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487755 T6641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487755 T6655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487755 T6609 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487765 T6587 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1487767 T6674 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a4ece4e name:ZooKeeperConnection Watcher:127.0.0.1:39032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1487767 T6587 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1487769 T6587 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1487777 T6603 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&maxShardsPerNode=2&name=delDeadColl&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 1487778 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487779 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487779 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487779 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487780 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487780 T6611 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delDeadColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1487781 T6675 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delDeadColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1487783 T6675 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1487783 T6675 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delDeadColl 
   [junit4]   2> 1487783 T6675 oascc.SolrZkClient.makePath makePath: /collections/delDeadColl
   [junit4]   2> 1487786 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487786 T6610 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1487787 T6610 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delDeadColl
   [junit4]   2> 1487787 T6610 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delDeadColl with shards [shard1, shard2]
   [junit4]   2> 1487788 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487889 T6627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6609 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6669 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487889 T6674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1487986 T6675 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delDeadColl, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1487986 T6675 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard1_replica1 as part of slice shard1 of collection delDeadColl on 127.0.0.1:40787_wllr%2Fl
   [junit4]   2> 1487990 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487990 T6675 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard1_replica2 as part of slice shard1 of collection delDeadColl on 127.0.0.1:41494_wllr%2Fl
   [junit4]   2> 1487990 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487991 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487993 T6610 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1487994 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1487994 T6675 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard2_replica1 as part of slice shard2 of collection delDeadColl on 127.0.0.1:38248_wllr%2Fl
   [junit4]   2> 1487994 T6610 oasc.Overseer$ClusterStateUpdater.createReplica createReplica() {
   [junit4]   2> 	  "operation":"ADDREPLICA",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"delDeadColl_shard1_replica1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40787/wllr/l"} 
   [junit4]   2> 1487999 T6675 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard2_replica2 as part of slice shard2 of collection delDeadColl on 127.0.0.1:41592_wllr%2Fl
   [junit4]   2> 1488003 T6610 oasc.Overseer$ClusterStateUpdater.createReplica createReplica() {
   [junit4]   2> 	  "operation":"ADDREPLICA",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"delDeadColl_shard1_replica2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41494/wllr/l"} 
   [junit4]   2> 1488005 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488006 T6610 oasc.Overseer$ClusterStateUpdater.createReplica createReplica() {
   [junit4]   2> 	  "operation":"ADDREPLICA",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "core":"delDeadColl_shard2_replica1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38248/wllr/l"} 
   [junit4]   2> 1488008 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488010 T6610 oasc.Overseer$ClusterStateUpdater.createReplica createReplica() {
   [junit4]   2> 	  "operation":"ADDREPLICA",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "core":"delDeadColl_shard2_replica2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41592/wllr/l"} 
   [junit4]   2> 1488018 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488110 T6649 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf1&name=delDeadColl_shard1_replica1&action=CREATE&numShards=2&collection=delDeadColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1488121 T6627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6664 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&name=delDeadColl_shard1_replica2&action=CREATE&numShards=2&collection=delDeadColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1488122 T6620 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf1&name=delDeadColl_shard2_replica2&action=CREATE&numShards=2&collection=delDeadColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1488121 T6609 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6669 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488121 T6641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488122 T6604 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&name=delDeadColl_shard2_replica1&action=CREATE&numShards=2&collection=delDeadColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1488214 T6649 oasc.ZkController.publish publishing core=delDeadColl_shard1_replica1 state=down collection=delDeadColl
   [junit4]   2> 1488216 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488216 T6649 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard1_replica1
   [junit4]   2> 1488217 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488217 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488218 T6610 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1488219 T6610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40787/wllr/l",
   [junit4]   2> 	  "core":"delDeadColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40787_wllr%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1488223 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488228 T6664 oasc.ZkController.publish publishing core=delDeadColl_shard1_replica2 state=down collection=delDeadColl
   [junit4]   2> 1488230 T6604 oasc.ZkController.publish publishing core=delDeadColl_shard2_replica1 state=down collection=delDeadColl
   [junit4]   2> 1488230 T6620 oasc.ZkController.publish publishing core=delDeadColl_shard2_replica2 state=down collection=delDeadColl
   [junit4]   2> 1488231 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488231 T6664 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard1_replica2
   [junit4]   2> 1488232 T6604 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard2_replica1
   [junit4]   2> 1488232 T6620 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard2_replica2
   [junit4]   2> 1488232 T6610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41494/wllr/l",
   [junit4]   2> 	  "core":"delDeadColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41494_wllr%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1488234 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488235 T6610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38248/wllr/l",
   [junit4]   2> 	  "core":"delDeadColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38248_wllr%2Fl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node3"}
   [junit4]   2> 1488238 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488239 T6610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41592/wllr/l",
   [junit4]   2> 	  "core":"delDeadColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41592_wllr%2Fl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node4"}
   [junit4]   2> 1488242 T6609 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1488344 T6627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6669 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6609 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1488344 T6614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1489218 T6649 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1489219 T6649 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1489219 T6649 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1489220 T6649 oascc.ZkStateReader.readConfigName path=/collections/delDeadColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1489221 T6649 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/'
   [junit4]   2> 1489237 T6604 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1489237 T6620 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1489237 T6664 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1489238 T6620 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1489238 T6664 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1489238 T6604 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1489239 T6664 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1489239 T6620 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1489239 T6604 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1489240 T6664 oascc.ZkStateReader.readConfigName path=/collections/delDeadColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1489240 T6664 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/'
   [junit4]   2> 1489240 T6604 oascc.ZkStateReader.readConfigName path=/collections/delDeadColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1489240 T6620 oascc.ZkStateReader.readConfigName path=/collections/delDeadColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1489241 T6604 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/'
   [junit4]   2> 1489241 T6620 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-003/delDeadColl_shard2_replica2/'
   [junit4]   2> 1489253 T6649 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1489270 T6664 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1489275 T6649 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1489277 T6649 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1489278 T6620 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1489286 T6604 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1489287 T6649 oass.IndexSchema.readSchema [delDeadColl_shard1_replica1] Schema name=test
   [junit4]   2> 1489288 T6664 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1489290 T6664 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1489308 T6620 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1489308 T6664 oass.IndexSchema.readSchema [delDeadColl_shard1_replica2] Schema name=test
   [junit4]   2> 1489309 T6620 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1489312 T6604 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1489325 T6604 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1489326 T6620 oass.IndexSchema.readSchema [delDeadColl_shard2_replica2] Schema name=test
   [junit4]   2> 1489339 T6604 oass.IndexSchema.readSchema [delDeadColl_shard2_replica1] Schema name=test
   [junit4]   2> 1489537 T6604 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1489539 T6664 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1489543 T6664 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1489543 T6604 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1489544 T6649 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1489545 T6664 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1489550 T6604 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1489553 T6649 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1489555 T6649 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1489556 T6664 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489559 T6604 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489559 T6664 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489561 T6664 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1489561 T6604 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489562 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1489563 T6649 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489563 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1489563 T6664 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1489564 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1489564 T6604 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1489564 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1489564 T6649 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489565 T6604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1489565 T6664 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard1_replica2' using configuration from collection delDeadColl
   [junit4]   2> 1489565 T6604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1489566 T6664 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1489566 T6604 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1489566 T6620 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1489566 T6664 oasc.SolrCore.<init> [delDeadColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/data/
   [junit4]   2> 1489566 T6649 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1489567 T6664 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e7a1db2
   [junit4]   2> 1489567 T6604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1489568 T6604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1489568 T6649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1489569 T6649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1489569 T6649 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1489569 T6604 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard2_replica1' using configuration from collection delDeadColl
   [junit4]   2> 1489568 T6664 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/data
   [junit4]   2> 1489570 T6649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1489570 T6620 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1489570 T6604 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1489570 T6649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1489570 T6664 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/data/index/
   [junit4]   2> 1489571 T6649 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard1_replica1' using configuration from collection delDeadColl
   [junit4]   2> 1489571 T6604 oasc.SolrCore.<init> [delDeadColl_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/data/
   [junit4]   2> 1489572 T6620 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1489572 T6604 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e7a1db2
   [junit4]   2> 1489572 T6649 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1489571 T6664 oasc.SolrCore.initIndex WARN [delDeadColl_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1489573 T6604 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/data
   [junit4]   2> 1489573 T6664 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-006/delDeadColl_shard1_replica2/data/index
   [junit4]   2> 1489573 T6649 oasc.SolrCore.<init> [delDeadColl_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/data/
   [junit4]   2> 1489574 T6649 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e7a1db2
   [junit4]   2> 1489574 T6664 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=33.3740234375, floorSegmentMB=1.2861328125, forceMergeDeletesPctAllowed=9.793873557357895, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1489575 T6649 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/data
   [junit4]   2> 1489574 T6604 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/data/index/
   [junit4]   2> 1489575 T6604 oasc.SolrCore.initIndex WARN [delDeadColl_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1489575 T6664 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30dce275 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a6440b5),segFN=segments_1,generation=1}
   [junit4]   2> 1489576 T6664 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1489575 T6649 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/data/index/
   [junit4]   2> 1489576 T6604 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-002/delDeadColl_shard2_replica1/data/index
   [junit4]   2> 1489577 T6649 oasc.SolrCore.initIndex WARN [delDeadColl_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1489577 T6604 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=33.3740234375, floorSegmentMB=1.2861328125, forceMergeDeletesPctAllowed=9.793873557357895, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1489577 T6649 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.DeleteInactiveReplicaTest-88BC324188A1A4A9-001/tempDir-005/delDeadColl_shard1_replica1/data/index
   [junit4]   2> 1489578 T6604 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3bdca082 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4adffc7c),segFN=segments_1,generation=1}
   [junit4]   2> 1489578 T6604 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1489578 T6649 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=33.3740234375, floorSegmentMB=1.2861328125, forceMergeDeletesPctAllowed=9.793873557357895, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1489579 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1489579 T6649 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ba18c2b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@70a44d1e),segFN=segments_1,generation=1}
   [junit4]   2> 1489580 T6620 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1489580 T6649 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1489579 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1489580 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1489581 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1489581 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1489581 T6604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1489581 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1489581 T6604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1489582 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-

[...truncated too long message...]

HostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 1537963 T6671 oaz.ClientCnxn$SendThread.run WARN Session 0x1469be86194000d for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 65790 T6586 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=el_CY, timezone=Etc/GMT-0
   [junit4]   2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=3,free=182761640,total=360497152
   [junit4]   2> NOTE: All tests run in this JVM: [TestSchemaResource, CoreMergeIndexesAdminHandlerTest, FieldAnalysisRequestHandlerTest, FileUtilsTest, PluginInfoTest, DocValuesMultiTest, TestSolrCoreProperties, DOMUtilTest, TestPartialUpdateDeduplication, TestReplicationHandlerBackup, TestFastOutputStream, DistributedQueryComponentCustomSortTest, TestExpandComponent, RemoteQueryErrorTest, TestReload, TestCloudManagedSchemaAddFields, TestDistributedSearch, TestFieldCollectionResource, TestHighFrequencyDictionaryFactory, DistribCursorPagingTest, TestDocumentBuilder, TestInitQParser, TestSimpleQParserPlugin, TestFoldingMultitermQuery, DistributedExpandComponentTest, HdfsCollectionsAPIDistributedZkTest, IndexSchemaRuntimeFieldTest, HdfsChaosMonkeySafeLeaderTest, TestSolrXMLSerializer, DateRangeFieldTest, PingRequestHandlerTest, TestCloudManagedSchema, TestCSVResponseWriter, QueryResultKeyTest, TestIndexingPerformance, TestMultiCoreConfBootstrap, AbstractAnalyticsFacetTest, TestSolr4Spatial, MultiTermTest, TestOrdValues, TestFiltering, ZkSolrClientTest, DirectUpdateHandlerTest, DisMaxRequestHandlerTest, IndexBasedSpellCheckerTest, SolrRequestParserTest, TestOmitPositions, BadCopyFieldTest, TestShardHandlerFactory, TestChildDocTransformer, SolrCoreCheckLockOnStartupTest, TestImplicitCoreProperties, CoreContainerCoreInitFailuresTest, CurrencyFieldXmlFileTest, StatsComponentTest, SuggesterWFSTTest, TestStressRecovery, StressHdfsTest, TestManagedSchemaFieldResource, TestDynamicFieldResource, DistributedSuggestComponentTest, DateFieldTest, TestStressLucene, CurrencyFieldOpenExchangeTest, TestSchemaNameResource, TestCollationField, TestXIncludeConfig, TestStandardQParsers, TestDistributedGrouping, TestHashPartitioner, TestAnalyzeInfixSuggestions, JSONWriterTest, CoreAdminRequestStatusTest, NoFacetTest, BasicDistributedZk2Test, TestComponentsName, RequiredFieldsTest, AlternateDirectoryTest, TestIBSimilarityFactory, DateMathParserTest, DebugComponentTest, TestRecovery, BadIndexSchemaTest, TestArbitraryIndexDir, TestPhraseSuggestions, RequestHandlersTest, ZkNodePropsTest, TermVectorComponentDistributedTest, TestSearcherReuse, DocumentAnalysisRequestHandlerTest, TestCoreDiscovery, HdfsSyncSliceTest, TestMaxScoreQueryParser, OutputWriterTest, AbstractAnalyticsStatsTest, FullSolrCloudDistribCmdsTest, MinimalSchemaTest, NumericFieldsTest, TestDFRSimilarityFactory, TestMiniSolrCloudCluster, ResourceLoaderTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, TestReplicationHandler, TestZkChroot, ZkCLITest, AutoCommitTest, TestStressReorder, TestStressVersions, HardAutoCommitTest, DistributedTermsComponentTest, SpellCheckComponentTest, SolrCmdDistributorTest, ConvertedLegacyTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, HighlighterTest, TestExtendedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, TestCSVLoader, TestPseudoReturnFields, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, TestRemoteStreaming, DistanceFunctionTest, SolrInfoMBeanTest, TestSurroundQueryParser, TestQueryUtils, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestQueryTypes, XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, SolrPluginUtilsTest, TestJmxIntegration, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, TestBinaryResponseWriter, SOLR749Test, UpdateParamsTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, TestBinaryField, TestFuzzyAnalyzedSuggestions, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, PreAnalyzedFieldTest, RAMDirectoryFactoryTest, TestSolrJ, TestUtils, SystemInfoHandlerTest, CursorPagingTest, FieldFacetExtrasTest, RangeFacetTest, AliasIntegrationTest, CollectionsAPIAsyncDistributedZkTest, DeleteInactiveReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteInactiveReplicaTest -Dtests.seed=88BC324188A1A4A9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el_CY -Dtests.timezone=Etc/GMT-0 -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | DeleteInactiveReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:451)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   [junit4]    >    1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    2) Thread[id=6672, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-EventThread, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=6671, name=TEST-DeleteInactiveReplicaTest.testDistribSearch-seed#[88BC324188A1A4A9]-SendThread(127.0.0.1:39032), state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([88BC324188A1A4A9]:0)
   [junit4] Completed on J1 in 65.82s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 203 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 416 suites, 1703 tests, 3 suite-level errors, 1 error, 216 ignored (16 assumptions)

Total time: 62 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message