lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b10) - Build # 9904 - Still Failing!
Date Tue, 25 Mar 2014 18:40:42 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9904/
Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
	at __randomizedtesting.SeedInfo.seed([E6F891076BD51336:58E98928CAF8644]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 45 more




Build Log:
[...truncated 11628 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 30306 T143 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=17 numCloses=17
   [junit4]   2> 30306 T143 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 30307 T143 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 30310 T143 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 30313 T143 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395772696955
   [junit4]   2> 30316 T143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 30318 T144 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 30418 T143 oasc.ZkTestServer.run start zk server on port:52972
   [junit4]   2> 30457 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 111109 T147 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 80633ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 111111 T143 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 111129 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 111131 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 111132 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 111133 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 111136 T143 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52972 52972
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=E6F891076BD51336 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=WET -Dtests.file.encoding=UTF-8
   [junit4] ERROR   81.0s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E6F891076BD51336:58E98928CAF8644]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
   [junit4]    > 	... 45 more
   [junit4]   2> 111334 T143 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395772777976
   [junit4]   2> 111335 T143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 111335 T151 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 111435 T143 oasc.ZkTestServer.run start zk server on port:40963
   [junit4]   2> 111436 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 111448 T157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f03c14 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 111449 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 111450 T143 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 111477 T152 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8a65fa0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 111477 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 111479 T159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63f864 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 111479 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 111482 T143 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 111487 T143 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 111490 T143 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 111492 T143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 111495 T143 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 111495 T143 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 111501 T143 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 111501 T143 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 111507 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 111507 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 111508 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 111508 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 111513 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 111513 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 111514 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 111514 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 111515 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 111523 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 111524 T161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d2d985 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 111525 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 111594 T143 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 111598 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 111600 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:47295
   [junit4]   2> 111602 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 111602 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 111602 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170
   [junit4]   2> 111603 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/'
   [junit4]   2> 111618 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/solr.xml
   [junit4]   2> 111669 T143 oasc.CoreContainer.<init> New CoreContainer 16423974
   [junit4]   2> 111669 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/]
   [junit4]   2> 111670 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 111670 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 111672 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 111672 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 111672 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 111675 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 111675 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 111675 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 111676 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
   [junit4]   2> 111685 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 111685 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 111687 T172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1760a96 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 111688 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 111693 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 111695 T174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9b695 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 111695 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 111705 T143 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 111710 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 111714 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 111718 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 111721 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 111728 T143 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 111731 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47295_
   [junit4]   2> 111733 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47295_
   [junit4]   2> 111736 T143 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 111799 T143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 111810 T143 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47295_
   [junit4]   2> 111810 T143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 111814 T143 oasc.Overseer.start Overseer (id=91473364777566212-127.0.0.1:47295_-n_0000000000) starting
   [junit4]   2> 111825 T143 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 111857 T176 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 111858 T143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 111859 T176 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 111865 T143 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 111867 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 111873 T175 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 111879 T177 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 111880 T177 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 111883 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 111883 T177 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 111886 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 111887 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47295",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47295_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 111887 T175 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 111893 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 111897 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 111901 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 112884 T177 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 112884 T177 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 112885 T177 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 112885 T177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 112887 T177 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 112887 T177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/collection1/'
   [junit4]   2> 112926 T177 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 112951 T177 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 112952 T177 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 112960 T177 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 112989 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 112995 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 113000 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 113005 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 113296 T177 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 113298 T177 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 113300 T177 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 113300 T177 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 113303 T177 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 113303 T177 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 113303 T177 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/
   [junit4]   2> 113304 T177 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 113304 T177 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data
   [junit4]   2> 113304 T177 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index/
   [junit4]   2> 113304 T177 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 113305 T177 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index
   [junit4]   2> 113306 T177 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2be308 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 113306 T177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 113307 T177 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 113307 T177 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 113307 T177 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 113309 T177 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 113311 T177 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 113313 T177 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 113313 T177 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 113314 T177 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2be308 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 113315 T177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 113315 T177 oass.SolrIndexSearcher.<init> Opening Searcher@1640c79[collection1] main
   [junit4]   2> 113315 T177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 113317 T177 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 113317 T177 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 113320 T177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 113320 T177 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 113321 T177 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 113322 T177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 113322 T177 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 113322 T177 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 113323 T177 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 113323 T178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1640c79[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 113323 T177 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 113324 T181 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:47295 collection:control_collection shard:shard1
   [junit4]   2> 113325 T143 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 113325 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 113331 T181 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 113331 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 113333 T183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14747a3 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 113333 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 113334 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 113339 T143 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 113339 T181 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 113342 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113343 T181 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 113343 T181 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 113343 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@33e9df url=https://127.0.0.1:47295/collection1 node=127.0.0.1:47295_ C24_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:47295, core=collection1, node_name=127.0.0.1:47295_}
   [junit4]   2> 113344 T181 C24 P47295 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:47295/collection1/
   [junit4]   2> 113345 T181 C24 P47295 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 113346 T181 C24 P47295 oasc.SyncStrategy.syncToMe https://127.0.0.1:47295/collection1/ has no replicas
   [junit4]   2> 113346 T181 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:47295/collection1/ shard1
   [junit4]   2> 113349 T181 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 113355 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113362 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113366 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113398 T143 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 113399 T143 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 113402 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 113403 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35092
   [junit4]   2> 113405 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 113405 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 113406 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980
   [junit4]   2> 113406 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/'
   [junit4]   2> 113420 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/solr.xml
   [junit4]   2> 113450 T143 oasc.CoreContainer.<init> New CoreContainer 6608415
   [junit4]   2> 113450 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/]
   [junit4]   2> 113451 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 113452 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 113452 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 113452 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 113454 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 113456 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 113457 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 113457 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 113457 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
   [junit4]   2> 113458 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 113458 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 113461 T194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d31d61 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 113461 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 113463 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 113464 T196 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1353d85 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 113464 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 113467 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 113467 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 113478 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 113513 T181 oasc.ZkController.register We are https://127.0.0.1:47295/collection1/ and leader is https://127.0.0.1:47295/collection1/
   [junit4]   2> 113514 T181 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:47295
   [junit4]   2> 113514 T181 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 113514 T181 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 113514 T181 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 113515 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113515 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113516 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113516 T181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 113517 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 113518 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47295",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47295_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 113521 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 113623 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 113623 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 113624 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 114483 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35092_
   [junit4]   2> 114485 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35092_
   [junit4]   2> 114488 T174 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 114488 T196 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 114488 T183 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 114495 T197 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 114496 T197 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 114497 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 114497 T197 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 114497 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 114498 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 114500 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 114500 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35092",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35092_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 114501 T175 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 114501 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 114503 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 114607 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 114607 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 114607 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 115498 T197 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 115498 T197 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 115499 T197 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 115499 T197 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 115500 T197 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 115501 T197 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/collection1/'
   [junit4]   2> 115539 T197 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 115573 T197 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 115574 T197 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 115586 T197 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 115844 T197 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 115846 T197 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 115848 T197 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 115848 T197 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 115849 T197 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 115849 T197 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 115849 T197 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/
   [junit4]   2> 115849 T197 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 115850 T197 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1
   [junit4]   2> 115850 T197 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index/
   [junit4]   2> 115850 T197 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 115850 T197 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index
   [junit4]   2> 115852 T197 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7c7f14 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 115852 T197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 115853 T197 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 115853 T197 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 115853 T197 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 115855 T197 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 115857 T197 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 115860 T197 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 115860 T197 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 115861 T197 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7c7f14 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 115861 T197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 115861 T197 oass.SolrIndexSearcher.<init> Opening Searcher@127a21a[collection1] main
   [junit4]   2> 115861 T197 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 115862 T197 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 115863 T197 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 115863 T197 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 115863 T197 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 115864 T197 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 115867 T197 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 115868 T197 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 115868 T197 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 115868 T197 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 115870 T198 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@127a21a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 115870 T197 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 115871 T201 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35092 collection:collection1 shard:shard1
   [junit4]   2> 115872 T143 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 115872 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 115875 T201 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 115883 T201 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 115886 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 115886 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 115886 T201 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 115886 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 115886 T201 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@1883f39 url=https://127.0.0.1:35092/collection1 node=127.0.0.1:35092_ C25_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:35092, core=collection1, node_name=127.0.0.1:35092_}
   [junit4]   2> 115887 T201 C25 P35092 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35092/collection1/
   [junit4]   2> 115887 T201 C25 P35092 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 115888 T201 C25 P35092 oasc.SyncStrategy.syncToMe https://127.0.0.1:35092/collection1/ has no replicas
   [junit4]   2> 115888 T201 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35092/collection1/ shard1
   [junit4]   2> 115888 T201 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 115889 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 115895 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 115901 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 115937 T143 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 115938 T143 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 115942 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 115943 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50957
   [junit4]   2> 115945 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 115946 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 115946 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515
   [junit4]   2> 115946 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/'
   [junit4]   2> 115958 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/solr.xml
   [junit4]   2> 115988 T143 oasc.CoreContainer.<init> New CoreContainer 8610917
   [junit4]   2> 115988 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/]
   [junit4]   2> 115989 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 115990 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 115990 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 115990 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 115992 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 115994 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 115994 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 115995 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 115995 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
   [junit4]   2> 115995 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 115996 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115998 T212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1182ea8 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115998 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 116000 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 116107 T214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8b11e name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 116107 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 116109 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 116109 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 116109 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 116115 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 116147 T201 oasc.ZkController.register We are https://127.0.0.1:35092/collection1/ and leader is https://127.0.0.1:35092/collection1/
   [junit4]   2> 116147 T201 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35092
   [junit4]   2> 116147 T201 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 116147 T201 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 116147 T201 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 116149 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 116149 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 116149 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 116149 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 116150 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 116151 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35092",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35092_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 116154 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 116256 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 116256 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 116256 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 116256 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 117119 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50957_
   [junit4]   2> 117121 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50957_
   [junit4]   2> 117126 T196 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 117126 T214 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 117126 T174 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 117127 T183 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 117141 T215 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 117141 T215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 117144 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 117144 T215 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 117145 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 117146 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 117148 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 117149 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50957",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50957_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 117150 T175 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 117151 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 117155 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 117259 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 117260 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 117260 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 117259 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118145 T215 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 118146 T215 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 118147 T215 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 118148 T215 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 118150 T215 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 118150 T215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/collection1/'
   [junit4]   2> 118182 T215 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 118221 T215 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 118223 T215 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 118230 T215 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 118474 T215 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 118477 T215 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 118478 T215 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 118479 T215 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 118479 T215 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 118479 T215 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 118480 T215 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/
   [junit4]   2> 118480 T215 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 118480 T215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2
   [junit4]   2> 118481 T215 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index/
   [junit4]   2> 118481 T215 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 118481 T215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index
   [junit4]   2> 118482 T215 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@c78e3a lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 118482 T215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 118483 T215 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 118485 T215 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 118485 T215 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 118485 T215 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 118487 T215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 118490 T215 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 118490 T215 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 118491 T215 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@c78e3a lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 118491 T215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 118491 T215 oass.SolrIndexSearcher.<init> Opening Searcher@1cc3776[collection1] main
   [junit4]   2> 118492 T215 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 118493 T215 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 118494 T215 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 118494 T215 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 118495 T215 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 118495 T215 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 118496 T215 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 118496 T215 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 118496 T215 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 118496 T215 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 118497 T216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1cc3776[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 118497 T215 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 118498 T219 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50957 collection:collection1 shard:shard2
   [junit4]   2> 118498 T143 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 118498 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 118498 T219 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 118503 T219 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 118505 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118505 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118505 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118505 T219 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 118506 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C26 name=collection1 org.apache.solr.core.SolrCore@1f3f11c url=https://127.0.0.1:50957/collection1 node=127.0.0.1:50957_ C26_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50957, core=collection1, node_name=127.0.0.1:50957_}
   [junit4]   2> 118506 T219 C26 P50957 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50957/collection1/
   [junit4]   2> 118506 T219 C26 P50957 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 118506 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 118506 T219 C26 P50957 oasc.SyncStrategy.syncToMe https://127.0.0.1:50957/collection1/ has no replicas
   [junit4]   2> 118507 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50957/collection1/ shard2
   [junit4]   2> 118507 T219 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 118509 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118511 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118514 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118550 T143 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 118550 T143 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 118556 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 118558 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49088
   [junit4]   2> 118560 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 118560 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 118561 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141
   [junit4]   2> 118561 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/'
   [junit4]   2> 118583 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/solr.xml
   [junit4]   2> 118621 T143 oasc.CoreContainer.<init> New CoreContainer 3282466
   [junit4]   2> 118621 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/]
   [junit4]   2> 118622 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 118625 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 118625 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118625 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118625 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118625 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118625 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 118627 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 118627 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 118630 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 118631 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 118631 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 118631 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
   [junit4]   2> 118632 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 118632 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 118634 T230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f7da08 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118634 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 118638 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 118639 T232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19823ed name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118639 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 118649 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 118670 T219 oasc.ZkController.register We are https://127.0.0.1:50957/collection1/ and leader is https://127.0.0.1:50957/collection1/
   [junit4]   2> 118670 T219 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50957
   [junit4]   2> 118670 T219 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 118670 T219 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 118671 T219 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 118672 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118673 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118673 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118673 T219 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 118676 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 118677 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50957",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50957_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 118681 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 118784 T232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118785 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118784 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118784 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 118785 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 119654 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49088_
   [junit4]   2> 119657 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49088_
   [junit4]   2> 119661 T196 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 119661 T183 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 119661 T174 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 119662 T232 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 119662 T214 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 119675 T233 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 119675 T233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 119678 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 119678 T233 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 119678 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 119679 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 119682 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 119683 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49088",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49088_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 119684 T175 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 119684 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 119688 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 119792 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 119792 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 119792 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 119792 T232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 119792 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 120679 T233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 120680 T233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 120681 T233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 120681 T233 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 120683 T233 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 120684 T233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/collection1/'
   [junit4]   2> 120720 T233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 120746 T233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 120747 T233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 120755 T233 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 121005 T233 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 121007 T233 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 121009 T233 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 121009 T233 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 121010 T233 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collec

[...truncated too long message...]

faultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 129559 T253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 129561 T143 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 129561 T253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50957",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50957_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 129562 T143 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 129563 T143 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2;done=false>>]
   [junit4]   2> 129563 T143 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2
   [junit4]   2> 129564 T143 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index;done=false>>]
   [junit4]   2> 129564 T143 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index
   [junit4]   2> 129566 T214 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 129568 T214 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 129570 T183 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 129592 T143 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 129643 T143 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 49088
   [junit4]   2> 129644 T143 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=3282466
   [junit4]   2> 129644 T143 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 129645 T143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 129650 T183 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 129672 T143 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 129727 T143 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47295
   [junit4]   2> 129736 T143 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 129738 T143 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40963 40963
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 129887 T143 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 130975 T196 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 131984 T232 C27 P49088 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 131991 T232 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 131992 T232 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1a7bed0
   [junit4]   2> 131993 T232 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 131993 T232 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 131994 T232 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 131994 T232 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 131997 T232 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 131997 T232 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 131997 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3;done=false>>]
   [junit4]   2> 131998 T232 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3
   [junit4]   2> 131998 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3/index;done=false>>]
   [junit4]   2> 131998 T232 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3/index
   [junit4]   2> 131999 T232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 131999 T232 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 132000 T232 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 132001 T232 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {text=PostingsFormat(name=MockSep), _version_=PostingsFormat(name=FSTPulsing41), range_facet_sl=PostingsFormat(name=MockSep), range_facet_l=PostingsFormat(name=FSTPulsing41), multiDefault=PostingsFormat(name=NestedPulsing), intDefault=PostingsFormat(name=FSTPulsing41), timestamp=PostingsFormat(name=FSTPulsing41), range_facet_si=PostingsFormat(name=NestedPulsing), id=PostingsFormat(name=Lucene41VarGapFixedInterval), a_t=PostingsFormat(name=FSTPulsing41)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=hu_HU, timezone=WET
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=90386648,total=185597952
   [junit4]   2> NOTE: All tests run in this JVM: [SolrExampleStreamingBinaryTest, ModifiableSolrParamsTest, TestDocumentObjectBinder, LargeVolumeEmbeddedTest, ContentStreamTest, TestUpdateRequestCodec, LargeVolumeJettyTest, SolrExceptionTest, TermsResponseTest, DocumentAnalysisResponseTest, TestEmbeddedSolrServer, TestCoreAdmin, CloudSolrServerTest]
   [junit4] Completed on J0 in 102.62s, 2 tests, 1 error <<< FAILURES!

[...truncated 94 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 error

Total time: 64 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message