lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0) - Build # 9809 - Still Failing!
Date Thu, 27 Mar 2014 02:40:44 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9809/
Java: 32bit/jdk1.8.0 -client -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43915 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43915 within 45000 ms
	at __randomizedtesting.SeedInfo.seed([9D17868F778A4454:7E618F1A90F0D126]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43915 within 45000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 45 more




Build Log:
[...truncated 11758 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 15885 T124 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 15885 T124 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 15887 T124 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 15889 T124 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395887907226
   [junit4]   2> 15892 T124 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 15894 T125 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 15993 T124 oasc.ZkTestServer.run start zk server on port:43915
   [junit4]   2> 16027 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 61110 T124 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 87353 T128 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 71297ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 87369 T126 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145016721e50000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 87371 T126 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145016721e50002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 87371 T126 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145016721e50003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 87373 T124 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43915 43915
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=9D17868F778A4454 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_GB -Dtests.timezone=Australia/Currie -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   71.7s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43915 within 45000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9D17868F778A4454:7E618F1A90F0D126]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43915 within 45000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
   [junit4]    > 	... 45 more
   [junit4]   2> 87564 T124 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395887978900
   [junit4]   2> 87564 T124 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 87565 T132 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 87665 T124 oasc.ZkTestServer.run start zk server on port:57652
   [junit4]   2> 87666 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 87703 T138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aa9713 name:ZooKeeperConnection Watcher:127.0.0.1:57652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 87704 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 87705 T124 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 87725 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 87727 T140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@733307 name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 87727 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 87731 T124 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 87736 T124 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 87739 T124 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 87741 T124 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 87744 T124 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 87744 T124 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 87748 T124 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 87749 T124 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 87753 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 87754 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 87754 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 87754 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 87755 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 87755 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 87755 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 87756 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 87756 T124 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 87760 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 87761 T142 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1142e05 name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 87761 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 87832 T124 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 87984 T124 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 87988 T124 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48114
   [junit4]   2> 87990 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 87990 T124 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 87990 T124 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395887979100
   [junit4]   2> 87991 T124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395887979100/'
   [junit4]   2> 88004 T124 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395887979100/solr.xml
   [junit4]   2> 88014 T124 oasc.CoreContainer.<init> New CoreContainer 21629462
   [junit4]   2> 88014 T124 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395887979100/]
   [junit4]   2> 88015 T124 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 88015 T124 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 88016 T124 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 88016 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 88016 T124 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 88017 T124 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 88017 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 88017 T124 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 88017 T124 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 88020 T124 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 88020 T124 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 88021 T124 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 88021 T124 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57652/solr
   [junit4]   2> 88031 T124 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 88031 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 88034 T153 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8f5b59 name:ZooKeeperConnection Watcher:127.0.0.1:57652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 88034 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 88040 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 88041 T155 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@191b219 name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 88041 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 88048 T124 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 88055 T124 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 88059 T124 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 88064 T124 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 88067 T124 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 88081 T124 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 88083 T124 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48114_
   [junit4]   2> 88086 T124 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48114_
   [junit4]   2> 88089 T124 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 88094 T124 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 88102 T124 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48114_
   [junit4]   2> 88102 T124 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 88105 T124 oasc.Overseer.start Overseer (id=91480914585321476-127.0.0.1:48114_-n_0000000000) starting
   [junit4]   2> 88113 T124 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 88135 T157 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 88135 T124 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 88137 T157 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 88140 T124 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 88143 T124 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 88151 T156 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 88157 T158 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 88157 T158 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 88160 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 88160 T158 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 88162 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 88164 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:48114",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48114_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 88164 T156 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 88171 T156 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 88177 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 88181 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 89161 T158 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 89161 T158 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 89162 T158 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 89162 T158 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 89163 T158 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 89164 T158 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395887979100/collection1/'
   [junit4]   2> 89191 T158 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 89198 T158 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 89199 T158 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 89206 T158 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 89219 T158 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 89223 T158 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 89227 T158 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 89232 T158 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 89371 T158 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 89373 T158 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 89374 T158 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 89374 T158 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 89377 T158 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 89377 T158 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 89377 T158 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395887979100/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data/
   [junit4]   2> 89377 T158 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 89378 T158 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data
   [junit4]   2> 89378 T158 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data/index/
   [junit4]   2> 89378 T158 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 89378 T158 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data/index
   [junit4]   2> 89379 T158 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1cc018a lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 89380 T158 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 89381 T158 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 89381 T158 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 89381 T158 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 89381 T158 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 89381 T158 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 89382 T158 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 89382 T158 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 89382 T158 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 89383 T158 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 89385 T158 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 89385 T158 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 89385 T158 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1cc018a lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 89386 T158 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 89386 T158 oass.SolrIndexSearcher.<init> Opening Searcher@1136e60[collection1] main
   [junit4]   2> 89387 T159 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1136e60[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 89387 T158 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 89388 T162 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:48114 collection:control_collection shard:shard1
   [junit4]   2> 89388 T124 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 89389 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 89395 T162 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 89397 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 89398 T164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b9f8a8 name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 89399 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 89400 T124 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 89402 T162 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 89404 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89404 T124 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 89405 T162 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 89405 T162 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 89406 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@1c75a0 url=https://127.0.0.1:48114/collection1 node=127.0.0.1:48114_ C9_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:48114, core=collection1, node_name=127.0.0.1:48114_}
   [junit4]   2> 89406 T162 C9 P48114 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:48114/collection1/
   [junit4]   2> 89407 T162 C9 P48114 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 89407 T162 C9 P48114 oasc.SyncStrategy.syncToMe https://127.0.0.1:48114/collection1/ has no replicas
   [junit4]   2> 89407 T162 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:48114/collection1/ shard1
   [junit4]   2> 89409 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89409 T162 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 89418 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89422 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89454 T124 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 89456 T124 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 89461 T124 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 89463 T124 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35495
   [junit4]   2> 89465 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 89466 T124 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 89466 T124 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395887980741
   [junit4]   2> 89466 T124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395887980741/'
   [junit4]   2> 89479 T124 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395887980741/solr.xml
   [junit4]   2> 89488 T124 oasc.CoreContainer.<init> New CoreContainer 13855105
   [junit4]   2> 89489 T124 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395887980741/]
   [junit4]   2> 89489 T124 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 89490 T124 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 89490 T124 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 89490 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 89490 T124 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 89491 T124 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 89491 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 89491 T124 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 89492 T124 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 89494 T124 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 89495 T124 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 89495 T124 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 89495 T124 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57652/solr
   [junit4]   2> 89496 T124 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 89497 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 89499 T175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d481d name:ZooKeeperConnection Watcher:127.0.0.1:57652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 89499 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 89501 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 89503 T177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141ca36 name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 89503 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 89510 T124 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 89524 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 89524 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 89524 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 89569 T162 oasc.ZkController.register We are https://127.0.0.1:48114/collection1/ and leader is https://127.0.0.1:48114/collection1/
   [junit4]   2> 89569 T162 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:48114
   [junit4]   2> 89569 T162 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 89570 T162 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 89570 T162 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 89571 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89571 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89571 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89572 T162 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 89573 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 89574 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:48114",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48114_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 89577 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 89681 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 89681 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 89681 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 90513 T124 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35495_
   [junit4]   2> 90514 T124 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35495_
   [junit4]   2> 90517 T177 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 90517 T155 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 90517 T164 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 90523 T178 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 90523 T178 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 90526 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 90526 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 90526 T178 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 90526 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 90528 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 90528 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35495",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35495_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 90528 T156 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 90529 T156 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 90531 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 90633 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 90633 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 90634 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91527 T178 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 91527 T178 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 91528 T178 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 91528 T178 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 91529 T178 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 91529 T178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395887980741/collection1/'
   [junit4]   2> 91547 T178 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 91555 T178 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 91557 T178 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 91563 T178 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 91664 T178 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 91666 T178 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 91667 T178 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 91667 T178 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 91668 T178 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 91668 T178 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 91668 T178 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395887980741/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1/
   [junit4]   2> 91668 T178 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 91669 T178 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1
   [junit4]   2> 91669 T178 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1/index/
   [junit4]   2> 91669 T178 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 91669 T178 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1/index
   [junit4]   2> 91671 T178 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12c44f7 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 91671 T178 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 91672 T178 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 91672 T178 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 91672 T178 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 91672 T178 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 91672 T178 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 91673 T178 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 91673 T178 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 91673 T178 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 91675 T178 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 91676 T178 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 91676 T178 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 91676 T178 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12c44f7 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 91677 T178 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 91677 T178 oass.SolrIndexSearcher.<init> Opening Searcher@23c33c[collection1] main
   [junit4]   2> 91678 T179 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23c33c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 91678 T178 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 91679 T182 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35495 collection:collection1 shard:shard2
   [junit4]   2> 91679 T124 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 91679 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 91680 T182 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 91686 T182 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 91689 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91689 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91689 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91689 T182 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 91690 T182 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@1cb6385 url=https://127.0.0.1:35495/collection1 node=127.0.0.1:35495_ C10_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:35495, core=collection1, node_name=127.0.0.1:35495_}
   [junit4]   2> 91690 T182 C10 P35495 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35495/collection1/
   [junit4]   2> 91690 T182 C10 P35495 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 91690 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 91691 T182 C10 P35495 oasc.SyncStrategy.syncToMe https://127.0.0.1:35495/collection1/ has no replicas
   [junit4]   2> 91691 T182 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35495/collection1/ shard2
   [junit4]   2> 91692 T182 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 91694 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91697 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91702 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91733 T124 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 91734 T124 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 91739 T124 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 91741 T124 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:33618
   [junit4]   2> 91743 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 91743 T124 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 91744 T124 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395887983017
   [junit4]   2> 91744 T124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395887983017/'
   [junit4]   2> 91757 T124 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395887983017/solr.xml
   [junit4]   2> 91766 T124 oasc.CoreContainer.<init> New CoreContainer 7527772
   [junit4]   2> 91766 T124 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395887983017/]
   [junit4]   2> 91767 T124 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 91767 T124 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 91768 T124 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 91768 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 91768 T124 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 91768 T124 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 91769 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 91769 T124 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 91769 T124 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 91771 T124 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 91772 T124 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 91772 T124 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 91772 T124 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57652/solr
   [junit4]   2> 91773 T124 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 91773 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 91776 T193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154dd7d name:ZooKeeperConnection Watcher:127.0.0.1:57652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 91777 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 91780 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 91785 T195 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1765e02 name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 91785 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 91793 T124 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 91804 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91804 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91804 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91804 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91848 T182 oasc.ZkController.register We are https://127.0.0.1:35495/collection1/ and leader is https://127.0.0.1:35495/collection1/
   [junit4]   2> 91849 T182 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35495
   [junit4]   2> 91849 T182 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 91849 T182 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 91849 T182 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 91850 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91851 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91851 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91851 T182 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 91852 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 91853 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35495",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35495_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 91856 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 91958 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91958 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91958 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91959 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 92796 T124 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33618_
   [junit4]   2> 92797 T124 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33618_
   [junit4]   2> 92799 T155 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 92800 T164 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 92800 T195 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 92800 T177 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 92806 T196 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 92806 T196 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 92808 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 92808 T196 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 92808 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 92808 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 92809 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 92810 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33618",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33618_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 92810 T156 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 92810 T156 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 92812 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 92915 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92915 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92915 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92915 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 93808 T196 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 93809 T196 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 93810 T196 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 93810 T196 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 93811 T196 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 93811 T196 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395887983017/collection1/'
   [junit4]   2> 93831 T196 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 93837 T196 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 93838 T196 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 93844 T196 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 93967 T196 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 93969 T196 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 93970 T196 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 93971 T196 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 93971 T196 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 93971 T196 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 93971 T196 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395887983017/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/
   [junit4]   2> 93972 T196 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 93972 T196 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2
   [junit4]   2> 93972 T196 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index/
   [junit4]   2> 93973 T196 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 93973 T196 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index
   [junit4]   2> 93974 T196 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@74726d lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 93974 T196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 93975 T196 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 93975 T196 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 93975 T196 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 93976 T196 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 93976 T196 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 93976 T196 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 93976 T196 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 93977 T196 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 93978 T196 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 93979 T196 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 93979 T196 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 93980 T196 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@74726d lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 93980 T196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 93980 T196 oass.SolrIndexSearcher.<init> Opening Searcher@1b6ee2a[collection1] main
   [junit4]   2> 93982 T197 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b6ee2a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 93982 T196 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 93982 T200 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:33618 collection:collection1 shard:shard1
   [junit4]   2> 93983 T124 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 93983 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 93983 T200 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 93988 T200 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 93990 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93990 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93990 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93991 T200 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 93991 T200 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@1a4605b url=https://127.0.0.1:33618/collection1 node=127.0.0.1:33618_ C11_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:33618, core=collection1, node_name=127.0.0.1:33618_}
   [junit4]   2> 93991 T200 C11 P33618 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:33618/collection1/
   [junit4]   2> 93991 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 93991 T200 C11 P33618 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 93992 T200 C11 P33618 oasc.SyncStrategy.syncToMe https://127.0.0.1:33618/collection1/ has no replicas
   [junit4]   2> 93992 T200 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:33618/collection1/ shard1
   [junit4]   2> 93992 T200 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 93994 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93996 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93999 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94037 T124 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 94038 T124 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 94042 T124 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 94045 T124 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53993
   [junit4]   2> 94047 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 94048 T124 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 94048 T124 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395887985321
   [junit4]   2> 94048 T124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395887985321/'
   [junit4]   2> 94069 T124 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395887985321/solr.xml
   [junit4]   2> 94079 T124 oasc.CoreContainer.<init> New CoreContainer 878807
   [junit4]   2> 94080 T124 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395887985321/]
   [junit4]   2> 94080 T124 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 94081 T124 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 94081 T124 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 94081 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 94081 T124 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 94082 T124 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 94082 T124 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 94082 T124 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 94082 T124 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 94085 T124 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 94085 T124 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 94085 T124 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 94086 T124 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57652/solr
   [junit4]   2> 94086 T124 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 94087 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94094 T211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f72816 name:ZooKeeperConnection Watcher:127.0.0.1:57652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94095 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94097 T124 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94099 T213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c87cfa name:ZooKeeperConnection Watcher:127.0.0.1:57652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94099 T124 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94104 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94104 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94104 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94104 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94111 T124 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 94147 T200 oasc.ZkController.register We are https://127.0.0.1:33618/collection1/ and leader is https://127.0.0.1:33618/collection1/
   [junit4]   2> 94148 T200 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:33618
   [junit4]   2> 94148 T200 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 94148 T200 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 94148 T200 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 94149 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94149 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94149 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94150 T200 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 94151 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 94151 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33618",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33618_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 94154 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94256 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94256 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94256 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94256 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94256 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 95114 T124 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53993_
   [junit4]   2> 95115 T124 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53993_
   [junit4]   2> 95118 T155 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 95118 T213 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 95118 T195 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 95118 T177 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 95118 T164 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 95145 T214 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 95145 T214 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 95147 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95147 T214 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 95147 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95147 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95149 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 95149 T156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53993",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53993_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 95149 T156 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 95150 T156 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 95152 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95254 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95254 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95254 T164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95254 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95254 T177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 96147 T214 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 96147 T214 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 96148 T214 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 96148 T214 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 96149 T214 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 96150 T214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395887985321/collection1/'
   [junit4]   2> 96176 T214 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 96183 T214 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 96184 T214 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 96193 T214 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 96284 T214 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 96286 T214 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 96287 T214 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 96288 T214 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 96288 T214 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 96288 T214 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 96288 T214 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395887985321/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/
   [junit4]   2> 96289 T214 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 96289 T214 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3
   [junit4]   2> 96289 T214 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index/
   [junit4]   2> 96289 T214 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 96290 T214 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index
   [junit4]   2> 96291 T214 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9ecf49 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 96291 T214 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 96292 T214 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 96292 T214 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 96292 T214 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 96292 T214 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 96293 T214 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 96293 T214 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 96293 T214 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 96293 T214 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 96295 T214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 96296 T214 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 96296 T214 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 96296 T214 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9ecf49 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 96297 T214 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 96297 T214 oass.SolrIndexSearcher.<init> Opening Searcher@5a30a3[collection1] main
   [junit4]   2> 96298 T215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a30a3[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 96298 T214 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 96299 T218 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53993 collection:collection1 shard:shard2
   [junit4]   2> 96299 T124 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 96299 T124 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 96301 T124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 96301 T218 oasc.ZkController.register We are https://127.0.0.1:53993/collection1/ and leader is https://127.0.0.1:35495/collection1/
   [junit4]   2> 96302 T218 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53993
   [junit4]   2> 96302 T218 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 96302 T218 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> 96302 T124 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
   [junit4]   2> 96303 T124 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 96303 T124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@c4dcae url=https://127.0.0.1:53993/collection1 node=127.0.0.1:53993_ C12_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:53993, core=collection1, node_name=127.0.0.1:53993_}
   [junit4]   2> 96307 T219 C12 P53993 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 96307 T218 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 96307 T219 C12 P53993 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 96308 T219 C12 P53993 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 96308 T219 C12 P53993 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 96309 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96309 T155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired 

[...truncated too long message...]

ery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=421,transaction_logs_total_number=5}
   [junit4]   2> 103431 T233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 103431 T124 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 103431 T124 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 103431 T124 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 103431 T233 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33618",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33618_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 103433 T124 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 103433 T124 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 103434 T124 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2;done=false>>]
   [junit4]   2> 103434 T124 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2
   [junit4]   2> 103434 T124 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index;done=false>>]
   [junit4]   2> 103434 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103435 T124 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty2/index
   [junit4]   2> 103437 T164 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 103459 T124 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 103511 T124 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53993
   [junit4]   2> 103511 T124 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=878807
   [junit4]   2> 103512 T124 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 103512 T124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 103515 T164 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 103536 T124 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 103590 T124 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 48114
   [junit4]   2> 103591 T124 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 103592 T124 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57652 57652
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 103655 T124 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 104755 T155 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 104850 T177 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 104937 T195 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 105854 T213 C12 P53993 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 105858 T213 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 105858 T213 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@c4dcae
   [junit4]   2> 105859 T213 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=421,transaction_logs_total_number=5}
   [junit4]   2> 105859 T213 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 105859 T213 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 105859 T213 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 105861 T213 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 105861 T213 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 105861 T213 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3;done=false>>]
   [junit4]   2> 105861 T213 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3
   [junit4]   2> 105861 T213 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index;done=false>>]
   [junit4]   2> 105861 T213 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395887978900/jetty3/index
   [junit4]   2> 105862 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 105862 T213 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 105862 T213 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 105862 T213 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_GB, timezone=Australia/Currie
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=126859240,total=185073664
   [junit4]   2> NOTE: All tests run in this JVM: [TermsResponseTest, LargeVolumeBinaryJettyTest, JettyWebappTest, TestCoreAdmin, LargeVolumeEmbeddedTest, TestXMLEscaping, AnlysisResponseBaseTest, SolrParamTest, TestUpdateRequestCodec, SolrExampleXMLTest, ClientUtilsTest, CloudSolrServerTest]
   [junit4] Completed on J0 in 90.81s, 2 tests, 1 error <<< FAILURES!

[...truncated 85 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 49 suites, 284 tests, 1 error

Total time: 55 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message