lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9701 - Failure!
Date Fri, 07 Mar 2014 05:52:56 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9701/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CustomCollectionTest.testDistribSearch

Error Message:
collection already exists: testCreateShardRepFactor

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: collection already exists: testCreateShardRepFactor
	at __randomizedtesting.SeedInfo.seed([C400B2084417940A:45E63C103348F436]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:497)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:302)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1608)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1570)
	at org.apache.solr.cloud.CustomCollectionTest.testCreateShardRepFactor(CustomCollectionTest.java:433)
	at org.apache.solr.cloud.CustomCollectionTest.doTest(CustomCollectionTest.java:129)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10317 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> 198999 T1412 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 198999 T1412 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 199002 T1412 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-CustomCollectionTest-1394170219459
   [junit4]   2> 199003 T1412 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 199004 T1413 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 199104 T1412 oasc.ZkTestServer.run start zk server on port:49473
   [junit4]   2> 199105 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 199159 T1419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aad804 name:ZooKeeperConnection Watcher:127.0.0.1:49473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 199159 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 199159 T1412 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 199164 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 199171 T1421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@519d2 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 199171 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 199171 T1412 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 199174 T1412 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 199176 T1412 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 199179 T1412 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 199181 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 199181 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 199185 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 199186 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 199190 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 199191 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 199194 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 199195 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 199200 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 199200 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 199203 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 199203 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 199205 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 199206 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 199208 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 199209 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 199211 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 199211 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 199214 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 199214 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 199216 T1412 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 199216 T1412 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 199220 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 199222 T1423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@897359 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 199222 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 199479 T1412 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 199482 T1412 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 199485 T1412 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34864
   [junit4]   2> 199486 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 199487 T1412 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 199487 T1412 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680
   [junit4]   2> 199487 T1412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/'
   [junit4]   2> 199501 T1412 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/solr.xml
   [junit4]   2> 199547 T1412 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680
   [junit4]   2> 199548 T1412 oasc.CoreContainer.<init> New CoreContainer 8891992
   [junit4]   2> 199548 T1412 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/]
   [junit4]   2> 199549 T1412 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 199549 T1412 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 199550 T1412 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 199550 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 199550 T1412 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 199551 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 199551 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 199551 T1412 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 199551 T1412 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 199555 T1412 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 199556 T1412 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 199556 T1412 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 199556 T1412 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49473/solr
   [junit4]   2> 199556 T1412 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 199557 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 199559 T1434 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fb9c58 name:ZooKeeperConnection Watcher:127.0.0.1:49473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 199559 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 199562 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 199564 T1436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@222983 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 199564 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 199565 T1412 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 199568 T1412 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 199572 T1412 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 199575 T1412 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34864_
   [junit4]   2> 199575 T1412 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34864_
   [junit4]   2> 199578 T1412 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 199580 T1412 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 199588 T1412 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34864_
   [junit4]   2> 199588 T1412 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 199592 T1412 oasc.Overseer.start Overseer (id=91368339502596100-127.0.0.1:34864_-n_0000000000) starting
   [junit4]   2> 199596 T1412 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 199603 T1438 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 199604 T1412 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 199604 T1438 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 199609 T1412 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 199612 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 199617 T1437 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 199621 T1412 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680
   [junit4]   2> 199632 T1412 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/conf/
   [junit4]   2> 199743 T1412 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 199744 T1412 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 199744 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 199747 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 199748 T1441 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@703ade name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 199749 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 199750 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 199753 T1412 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 199990 T1412 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 199992 T1412 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 199995 T1412 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 199997 T1412 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50847
   [junit4]   2> 200000 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 200001 T1412 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 200001 T1412 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209
   [junit4]   2> 200002 T1412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/'
   [junit4]   2> 200015 T1412 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/solr.xml
   [junit4]   2> 200057 T1412 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209
   [junit4]   2> 200057 T1412 oasc.CoreContainer.<init> New CoreContainer 25103255
   [junit4]   2> 200057 T1412 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/]
   [junit4]   2> 200058 T1412 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 200059 T1412 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 200059 T1412 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 200059 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 200059 T1412 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 200060 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 200060 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 200060 T1412 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 200061 T1412 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 200064 T1412 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 200064 T1412 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 200064 T1412 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 200065 T1412 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49473/solr
   [junit4]   2> 200065 T1412 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 200066 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 200067 T1452 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55a6e6 name:ZooKeeperConnection Watcher:127.0.0.1:49473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 200067 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 200070 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 200072 T1454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@520e10 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 200072 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 200080 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 201085 T1412 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50847_
   [junit4]   2> 201086 T1412 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50847_
   [junit4]   2> 201088 T1436 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 201088 T1454 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 201089 T1441 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 201093 T1412 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209
   [junit4]   2> 201099 T1412 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/conf/
   [junit4]   2> 201206 T1412 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 201206 T1412 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 201207 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 201426 T1412 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 201426 T1412 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 201429 T1412 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 201430 T1412 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50377
   [junit4]   2> 201432 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 201432 T1412 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 201433 T1412 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664
   [junit4]   2> 201433 T1412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/'
   [junit4]   2> 201444 T1412 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/solr.xml
   [junit4]   2> 201499 T1412 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664
   [junit4]   2> 201499 T1412 oasc.CoreContainer.<init> New CoreContainer 25092420
   [junit4]   2> 201499 T1412 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/]
   [junit4]   2> 201501 T1412 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 201501 T1412 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 201501 T1412 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 201501 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 201502 T1412 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 201502 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 201502 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 201502 T1412 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 201503 T1412 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 201506 T1412 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 201507 T1412 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 201507 T1412 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 201507 T1412 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49473/solr
   [junit4]   2> 201508 T1412 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 201508 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 201509 T1466 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d4af0 name:ZooKeeperConnection Watcher:127.0.0.1:49473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 201509 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 201511 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 201512 T1468 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9d1789 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 201513 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 201516 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 202518 T1412 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50377_
   [junit4]   2> 202519 T1412 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50377_
   [junit4]   2> 202522 T1441 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 202522 T1454 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 202522 T1468 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 202522 T1436 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 202527 T1412 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664
   [junit4]   2> 202533 T1412 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/conf/
   [junit4]   2> 202646 T1412 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 202647 T1412 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 202647 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 202871 T1412 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 202872 T1412 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 202874 T1412 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 202875 T1412 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35010
   [junit4]   2> 202877 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 202878 T1412 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 202878 T1412 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104
   [junit4]   2> 202878 T1412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/'
   [junit4]   2> 202889 T1412 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/solr.xml
   [junit4]   2> 202930 T1412 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104
   [junit4]   2> 202930 T1412 oasc.CoreContainer.<init> New CoreContainer 12492017
   [junit4]   2> 202931 T1412 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/]
   [junit4]   2> 202932 T1412 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 202932 T1412 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 202932 T1412 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 202932 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 202933 T1412 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 202933 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 202933 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 202934 T1412 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 202934 T1412 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 202937 T1412 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 202938 T1412 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 202938 T1412 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 202938 T1412 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49473/solr
   [junit4]   2> 202938 T1412 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 202939 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 202940 T1480 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64b4f5 name:ZooKeeperConnection Watcher:127.0.0.1:49473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 202941 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 202943 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 202944 T1482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64e381 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 202944 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 202948 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 203950 T1412 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35010_
   [junit4]   2> 203951 T1412 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35010_
   [junit4]   2> 203954 T1441 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 203954 T1482 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 203954 T1454 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 203955 T1468 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 203954 T1436 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 203960 T1412 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104
   [junit4]   2> 203965 T1412 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/conf/
   [junit4]   2> 204057 T1412 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 204057 T1412 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 204058 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 204293 T1412 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 204293 T1412 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 204296 T1412 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 204297 T1412 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42989
   [junit4]   2> 204299 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 204299 T1412 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 204300 T1412 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515
   [junit4]   2> 204300 T1412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/'
   [junit4]   2> 204311 T1412 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/solr.xml
   [junit4]   2> 204356 T1412 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515
   [junit4]   2> 204357 T1412 oasc.CoreContainer.<init> New CoreContainer 16668337
   [junit4]   2> 204357 T1412 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/]
   [junit4]   2> 204358 T1412 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 204359 T1412 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 204359 T1412 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 204359 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 204360 T1412 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 204360 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 204360 T1412 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 204360 T1412 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 204361 T1412 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 204364 T1412 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 204365 T1412 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 204365 T1412 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 204365 T1412 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49473/solr
   [junit4]   2> 204365 T1412 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 204366 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 204367 T1494 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f5cbd1 name:ZooKeeperConnection Watcher:127.0.0.1:49473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 204368 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 204370 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 204371 T1496 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a91a7 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 204371 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 204375 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 205377 T1412 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42989_
   [junit4]   2> 205378 T1412 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42989_
   [junit4]   2> 205381 T1454 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 205382 T1482 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 205382 T1436 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 205382 T1496 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 205381 T1468 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 205381 T1441 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 205388 T1412 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515
   [junit4]   2> 205398 T1412 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/conf/
   [junit4]   2> 205518 T1412 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 205518 T1412 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 205519 T1412 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 205521 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 205522 T1499 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e96fa8 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 205523 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 205524 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 205528 T1412 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 205529 T1501 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@925131 name:ZooKeeperConnection Watcher:127.0.0.1:49473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 205529 T1412 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 205530 T1412 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 205598 T1491 oasha.CollectionsHandler.handleCreateAction Creating Collection : maxShardsPerNode=5&version=2&wt=javabin&action=CREATE&router.name=implicit&shards=a,b,c&name=implicitcoll0&replicationFactor=5
   [junit4]   2> 205600 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 205601 T1438 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"implicitcoll0",
   [junit4]   2> 	  "replicationFactor":"5",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b,c",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 205601 T1438 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"implicitcoll0",
   [junit4]   2> 	  "replicationFactor":"5",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b,c",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 205603 T1438 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 205603 T1438 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/implicitcoll0 
   [junit4]   2> 205603 T1438 oascc.SolrZkClient.makePath makePath: /collections/implicitcoll0
   [junit4]   2> 205607 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 205608 T1437 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 205609 T1437 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: implicitcoll0
   [junit4]   2> 205609 T1437 oasc.Overseer$ClusterStateUpdater.createCollection Create collection implicitcoll0 with shards [a, b, c]
   [junit4]   2> 205611 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 205612 T1454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1436 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1501 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205612 T1468 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 205707 T1438 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [a, b, c] , replicationFactor : 5
   [junit4]   2> 205708 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_a_replica1 as part of slice a of collection implicitcoll0 on 127.0.0.1:34864_
   [junit4]   2> 205709 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_a_replica2 as part of slice a of collection implicitcoll0 on 127.0.0.1:50847_
   [junit4]   2> 205710 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_a_replica3 as part of slice a of collection implicitcoll0 on 127.0.0.1:42989_
   [junit4]   2> 205712 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_a_replica4 as part of slice a of collection implicitcoll0 on 127.0.0.1:35010_
   [junit4]   2> 205719 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_a_replica5 as part of slice a of collection implicitcoll0 on 127.0.0.1:50377_
   [junit4]   2> 205720 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_b_replica1 as part of slice b of collection implicitcoll0 on 127.0.0.1:34864_
   [junit4]   2> 205725 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_b_replica2 as part of slice b of collection implicitcoll0 on 127.0.0.1:50847_
   [junit4]   2> 205732 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_b_replica3 as part of slice b of collection implicitcoll0 on 127.0.0.1:42989_
   [junit4]   2> 205734 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_b_replica4 as part of slice b of collection implicitcoll0 on 127.0.0.1:35010_
   [junit4]   2> 205747 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_b_replica5 as part of slice b of collection implicitcoll0 on 127.0.0.1:50377_
   [junit4]   2> 205810 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_c_replica1 as part of slice c of collection implicitcoll0 on 127.0.0.1:34864_
   [junit4]   2> 205836 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_c_replica2 as part of slice c of collection implicitcoll0 on 127.0.0.1:50847_
   [junit4]   2> 205859 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_c_replica3 as part of slice c of collection implicitcoll0 on 127.0.0.1:42989_
   [junit4]   2> 205876 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_c_replica4 as part of slice c of collection implicitcoll0 on 127.0.0.1:35010_
   [junit4]   2> 205879 T1438 oasc.OverseerCollectionProcessor.createCollection Creating shard implicitcoll0_c_replica5 as part of slice c of collection implicitcoll0 on 127.0.0.1:50377_
   [junit4]   2> 205907 T1450 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=a&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_a_replica2
   [junit4]   2> 205928 T1450 oasc.ZkController.publish publishing core=implicitcoll0_a_replica2 state=down collection=implicitcoll0
   [junit4]   2> 205929 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 205929 T1450 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 205931 T1437 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 205931 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50847",
   [junit4]   2> 	  "core":"implicitcoll0_a_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50847_",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 205933 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 205985 T1448 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=b&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_b_replica2
   [junit4]   2> 205987 T1448 oasc.ZkController.publish publishing core=implicitcoll0_b_replica2 state=down collection=implicitcoll0
   [junit4]   2> 205989 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 205989 T1448 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 205990 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50847",
   [junit4]   2> 	  "core":"implicitcoll0_b_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50847_",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 205995 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206008 T1489 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=a&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_a_replica3
   [junit4]   2> 206010 T1489 oasc.ZkController.publish publishing core=implicitcoll0_a_replica3 state=down collection=implicitcoll0
   [junit4]   2> 206011 T1489 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206012 T1430 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=a&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_a_replica1
   [junit4]   2> 206011 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206013 T1492 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=b&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_b_replica3
   [junit4]   2> 206015 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42989",
   [junit4]   2> 	  "core":"implicitcoll0_a_replica3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42989_",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206020 T1462 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=a&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_a_replica5
   [junit4]   2> 206020 T1430 oasc.ZkController.publish publishing core=implicitcoll0_a_replica1 state=down collection=implicitcoll0
   [junit4]   2> 206021 T1492 oasc.ZkController.publish publishing core=implicitcoll0_b_replica3 state=down collection=implicitcoll0
   [junit4]   2> 206022 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206022 T1475 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=a&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_a_replica4
   [junit4]   2> 206028 T1430 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206032 T1492 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206034 T1432 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=b&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_b_replica1
   [junit4]   2> 206035 T1462 oasc.ZkController.publish publishing core=implicitcoll0_a_replica5 state=down collection=implicitcoll0
   [junit4]   2> 206035 T1461 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=b&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_b_replica5
   [junit4]   2> 206036 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34864",
   [junit4]   2> 	  "core":"implicitcoll0_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34864_",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206036 T1475 oasc.ZkController.publish publishing core=implicitcoll0_a_replica4 state=down collection=implicitcoll0
   [junit4]   2> 206037 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206038 T1475 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206038 T1462 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206040 T1432 oasc.ZkController.publish publishing core=implicitcoll0_b_replica1 state=down collection=implicitcoll0
   [junit4]   2> 206040 T1461 oasc.ZkController.publish publishing core=implicitcoll0_b_replica5 state=down collection=implicitcoll0
   [junit4]   2> 206043 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206043 T1461 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206043 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42989",
   [junit4]   2> 	  "core":"implicitcoll0_b_replica3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42989_",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206043 T1432 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206046 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50377",
   [junit4]   2> 	  "core":"implicitcoll0_a_replica5",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50377_",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206043 T1476 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=b&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_b_replica4
   [junit4]   2> 206047 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206048 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35010",
   [junit4]   2> 	  "core":"implicitcoll0_a_replica4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35010_",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206050 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206051 T1476 oasc.ZkController.publish publishing core=implicitcoll0_b_replica4 state=down collection=implicitcoll0
   [junit4]   2> 206051 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50377",
   [junit4]   2> 	  "core":"implicitcoll0_b_replica5",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50377_",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206052 T1476 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206052 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206054 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34864",
   [junit4]   2> 	  "core":"implicitcoll0_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34864_",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206070 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206075 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35010",
   [junit4]   2> 	  "core":"implicitcoll0_b_replica4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35010_",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206076 T1490 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=c&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_c_replica3
   [junit4]   2> 206078 T1477 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=c&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_c_replica4
   [junit4]   2> 206079 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206082 T1490 oasc.ZkController.publish publishing core=implicitcoll0_c_replica3 state=down collection=implicitcoll0
   [junit4]   2> 206084 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206086 T1490 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206086 T1429 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=c&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_c_replica1
   [junit4]   2> 206087 T1477 oasc.ZkController.publish publishing core=implicitcoll0_c_replica4 state=down collection=implicitcoll0
   [junit4]   2> 206087 T1449 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=c&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_c_replica2
   [junit4]   2> 206087 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42989",
   [junit4]   2> 	  "core":"implicitcoll0_c_replica3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42989_",
   [junit4]   2> 	  "shard":"c",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206088 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206088 T1477 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206090 T1449 oasc.ZkController.publish publishing core=implicitcoll0_c_replica2 state=down collection=implicitcoll0
   [junit4]   2> 206091 T1464 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=implicitcoll0&shard=c&collection.configName=conf1&wt=javabin&qt=/admin/cores&numShards=3&action=CREATE&name=implicitcoll0_c_replica5
   [junit4]   2> 206091 T1449 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206091 T1429 oasc.ZkController.publish publishing core=implicitcoll0_c_replica1 state=down collection=implicitcoll0
   [junit4]   2> 206093 T1429 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206093 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35010",
   [junit4]   2> 	  "core":"implicitcoll0_c_replica4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35010_",
   [junit4]   2> 	  "shard":"c",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206093 T1464 oasc.ZkController.publish publishing core=implicitcoll0_c_replica5 state=down collection=implicitcoll0
   [junit4]   2> 206094 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206094 T1464 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 206096 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50847",
   [junit4]   2> 	  "core":"implicitcoll0_c_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50847_",
   [junit4]   2> 	  "shard":"c",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206098 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206098 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34864",
   [junit4]   2> 	  "core":"implicitcoll0_c_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34864_",
   [junit4]   2> 	  "shard":"c",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206100 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206101 T1437 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50377",
   [junit4]   2> 	  "core":"implicitcoll0_c_replica5",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50377_",
   [junit4]   2> 	  "shard":"c",
   [junit4]   2> 	  "collection":"implicitcoll0",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 206102 T1436 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 206244 T1454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1436 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1501 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1468 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206244 T1496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 206930 T1450 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_a_replica2
   [junit4]   2> 206931 T1450 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_a_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/implicitcoll0_a_replica2
   [junit4]   2> 206932 T1450 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 206933 T1450 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 206933 T1450 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 206934 T1450 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 206935 T1450 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/implicitcoll0_a_replica2/'
   [junit4]   2> 206980 T1450 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 206989 T1448 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_b_replica2
   [junit4]   2> 206991 T1448 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_b_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/implicitcoll0_b_replica2
   [junit4]   2> 206991 T1448 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 206992 T1448 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 206992 T1448 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 206993 T1448 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 206994 T1448 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/implicitcoll0_b_replica2/'
   [junit4]   2> 207012 T1489 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_a_replica3
   [junit4]   2> 207013 T1489 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_a_replica3' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/implicitcoll0_a_replica3
   [junit4]   2> 207014 T1489 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207015 T1489 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207015 T1489 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207016 T1489 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207017 T1489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/implicitcoll0_a_replica3/'
   [junit4]   2> 207028 T1430 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_a_replica1
   [junit4]   2> 207029 T1430 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_a_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/implicitcoll0_a_replica1
   [junit4]   2> 207030 T1430 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207030 T1430 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207031 T1430 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207032 T1430 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207032 T1430 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/implicitcoll0_a_replica1/'
   [junit4]   2> 207033 T1492 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_b_replica3
   [junit4]   2> 207034 T1492 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_b_replica3' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/implicitcoll0_b_replica3
   [junit4]   2> 207034 T1492 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207035 T1492 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207036 T1492 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207037 T1492 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207037 T1492 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/implicitcoll0_b_replica3/'
   [junit4]   2> 207038 T1475 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_a_replica4
   [junit4]   2> 207038 T1462 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_a_replica5
   [junit4]   2> 207039 T1475 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_a_replica4' using instanceDir: ./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/implicitcoll0_a_replica4
   [junit4]   2> 207039 T1475 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207039 T1462 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_a_replica5' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/implicitcoll0_a_replica5
   [junit4]   2> 207040 T1462 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207040 T1475 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207041 T1475 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207041 T1462 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207042 T1462 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207042 T1475 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207042 T1475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/implicitcoll0_a_replica4/'
   [junit4]   2> 207041 T1450 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 207043 T1462 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207043 T1461 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_b_replica5
   [junit4]   2> 207044 T1432 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_b_replica1
   [junit4]   2> 207043 T1462 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/implicitcoll0_a_replica5/'
   [junit4]   2> 207045 T1461 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_b_replica5' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/implicitcoll0_b_replica5
   [junit4]   2> 207045 T1461 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207045 T1432 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_b_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/implicitcoll0_b_replica1
   [junit4]   2> 207046 T1432 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207045 T1450 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 207047 T1432 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207047 T1432 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207048 T1461 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207051 T1461 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207051 T1448 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 207052 T1461 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207053 T1461 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/implicitcoll0_b_replica5/'
   [junit4]   2> 207049 T1432 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207053 T1476 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_b_replica4
   [junit4]   2> 207054 T1476 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_b_replica4' using instanceDir: ./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/implicitcoll0_b_replica4
   [junit4]   2> 207055 T1476 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207054 T1432 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/implicitcoll0_b_replica1/'
   [junit4]   2> 207056 T1476 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207056 T1476 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207060 T1476 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207060 T1476 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/implicitcoll0_b_replica4/'
   [junit4]   2> 207064 T1450 oass.IndexSchema.readSchema [implicitcoll0_a_replica2] Schema name=test
   [junit4]   2> 207094 T1449 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_c_replica2
   [junit4]   2> 207094 T1429 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_c_replica1
   [junit4]   2> 207094 T1477 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_c_replica4
   [junit4]   2> 207096 T1429 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_c_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-controljetty-1394170219680/implicitcoll0_c_replica1
   [junit4]   2> 207095 T1449 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_c_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty1-1394170220209/implicitcoll0_c_replica2
   [junit4]   2> 207096 T1449 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207095 T1464 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_c_replica5
   [junit4]   2> 207095 T1490 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for implicitcoll0_c_replica3
   [junit4]   2> 207098 T1489 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 207096 T1477 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_c_replica4' using instanceDir: ./org.apache.solr.cloud.CustomCollectionTest-jetty3-1394170223104/implicitcoll0_c_replica4
   [junit4]   2> 207098 T1477 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207096 T1429 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207098 T1490 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_c_replica3' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty4-1394170224515/implicitcoll0_c_replica3
   [junit4]   2> 207100 T1429 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207098 T1449 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207101 T1449 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207098 T1464 oasc.CoreContainer.create Creating SolrCore 'implicitcoll0_c_replica5' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CustomCollectionTest-jetty2-1394170221664/implicitcoll0_c_replica5
   [junit4]   2> 207102 T1464 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207100 T1429 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207100 T1490 oasc.ZkController.createCollectionZkNode Check for collection zkNode:implicitcoll0
   [junit4]   2> 207100 T1477 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207105 T1477 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207103 T1429 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 207102 T1464 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 207106 T1464 oascc.ZkStateReader.readConfigName Load collection config from:/collections/implicitcoll0
   [junit4]   2> 207102 T1449 oascc.ZkStateReader.readConfigName path=/collections/implicitcoll0 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2071

[...truncated too long message...]

eate(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 550957 T1496 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/implicitcoll2/leader_elect/a/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 550958 T1496 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/implicitcollwithShardField/leader_elect/b/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 550958 T1496 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/implicitcoll2/leader_elect/c/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 550958 T1496 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 550959 T1496 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/implicitcoll1/leader_elect/c/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=de_CH, timezone=Pacific/Norfolk
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=118798648,total=295174144
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrQueryParser, TestSuggestSpellingConverter, TestSystemIdResolver, TestLMJelinekMercerSimilarityFactory, HdfsLockFactoryTest, TriLevelCompositeIdRoutingTest, TestJoin, LegacyHTMLStripCharFilterTest, TestLMDirichletSimilarityFactory, DistributedQueryComponentOptimizationTest, TestSchemaSimilarityResource, CoreAdminCreateDiscoverTest, TestReplicationHandler, TestRealTimeGet, ScriptEngineTest, QueryEqualityTest, CustomCollectionTest]
   [junit4] Completed on J0 in 352.60s, 1 test, 1 error <<< FAILURES!

[...truncated 932 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 378 suites, 1618 tests, 1 error, 49 ignored (20 assumptions)

Total time: 50 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message