lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_05) - Build # 10370 - Still Failing!
Date Sat, 24 May 2014 17:56:24 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10370/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
	at __randomizedtesting.SeedInfo.seed([B7F016F133CCE3E5:361698E9449383D9]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:650)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1141)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1110)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:236)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11162 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/init-core-data-001
   [junit4]   2> 726072 T2472 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 726072 T2472 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 726075 T2472 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 726075 T2472 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 726076 T2473 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 726176 T2472 oasc.ZkTestServer.run start zk server on port:48431
   [junit4]   2> 726177 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 727517 T2476 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1338ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 727519 T2479 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e64067 name:ZooKeeperConnection Watcher:127.0.0.1:48431 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727520 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 727521 T2472 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 727527 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 727529 T2481 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bdab2 name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727529 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 727529 T2472 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 727532 T2472 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 727534 T2472 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 727535 T2472 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 727539 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 727539 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 727543 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 727544 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 727549 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 727549 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 727551 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 727552 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 727555 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 727555 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 727557 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 727558 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 727560 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 727561 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 727563 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 727564 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 727566 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 727567 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 727569 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 727569 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 727571 T2472 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 727571 T2472 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 727574 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 727575 T2483 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18bb850 name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727575 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 727822 T2472 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 727825 T2472 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 727828 T2472 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48726
   [junit4]   2> 727829 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 727829 T2472 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 727829 T2472 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002
   [junit4]   2> 727830 T2472 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/'
   [junit4]   2> 727843 T2472 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/solr.xml
   [junit4]   2> 727851 T2472 oasc.CoreContainer.<init> New CoreContainer 9832890
   [junit4]   2> 727852 T2472 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/]
   [junit4]   2> 727853 T2472 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 727853 T2472 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 727853 T2472 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 727853 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 727854 T2472 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 727854 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 727854 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 727855 T2472 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 727855 T2472 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 727857 T2472 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 727857 T2472 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 727858 T2472 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 727858 T2472 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48431/solr
   [junit4]   2> 727858 T2472 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 727859 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 727861 T2494 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aa9524 name:ZooKeeperConnection Watcher:127.0.0.1:48431 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727861 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 727863 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 727864 T2496 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@108e7b6 name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727864 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 727866 T2472 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 727868 T2472 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 727870 T2472 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 727872 T2472 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 727874 T2472 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 727876 T2472 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 727877 T2472 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48726_
   [junit4]   2> 727882 T2472 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48726_
   [junit4]   2> 727884 T2472 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 727885 T2472 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 727888 T2472 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48726_
   [junit4]   2> 727888 T2472 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 727890 T2472 oasc.Overseer.start Overseer (id=91812870482886660-127.0.0.1:48726_-n_0000000000) starting
   [junit4]   2> 727892 T2472 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 727900 T2498 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 727900 T2472 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 727903 T2472 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 727904 T2472 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 727906 T2497 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 727913 T2499 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 727913 T2499 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 727914 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727914 T2499 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 727915 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727916 T2497 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:48726",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48726_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 727916 T2497 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 727916 T2497 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 727918 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727919 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 728915 T2499 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 728915 T2499 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 728917 T2499 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 728917 T2499 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 728919 T2499 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 728919 T2499 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/collection1/'
   [junit4]   2> 728921 T2499 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 728921 T2499 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 728922 T2499 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 728957 T2499 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 728967 T2499 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 728968 T2499 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 728974 T2499 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 729093 T2499 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 729097 T2499 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 729098 T2499 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 729116 T2499 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 729119 T2499 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 729122 T2499 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 729123 T2499 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 729123 T2499 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 729123 T2499 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 729125 T2499 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 729125 T2499 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 729125 T2499 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 729126 T2499 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 729126 T2499 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/control/data/
   [junit4]   2> 729126 T2499 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43456
   [junit4]   2> 729127 T2499 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/control/data
   [junit4]   2> 729127 T2499 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/control/data/index/
   [junit4]   2> 729127 T2499 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 729128 T2499 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/control/data/index
   [junit4]   2> 729129 T2499 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=20.5791015625, floorSegmentMB=0.4296875, forceMergeDeletesPctAllowed=29.988170746820686, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19306883707246503
   [junit4]   2> 729130 T2499 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1c4c1bb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1319eb8),segFN=segments_1,generation=1}
   [junit4]   2> 729130 T2499 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 729135 T2499 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 729135 T2499 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 729136 T2499 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 729136 T2499 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 729137 T2499 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 729137 T2499 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 729138 T2499 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 729138 T2499 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 729138 T2499 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 729139 T2499 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 729140 T2499 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 729140 T2499 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 729141 T2499 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 729142 T2499 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 729142 T2499 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 729143 T2499 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 729144 T2499 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 729147 T2499 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 729149 T2499 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 729149 T2499 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 729149 T2499 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=67.328125, floorSegmentMB=2.181640625, forceMergeDeletesPctAllowed=20.852550126679752, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 729150 T2499 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1c4c1bb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1319eb8),segFN=segments_1,generation=1}
   [junit4]   2> 729150 T2499 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 729150 T2499 oass.SolrIndexSearcher.<init> Opening Searcher@4c48a9[collection1] main
   [junit4]   2> 729151 T2499 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 729152 T2499 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 729152 T2499 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 729153 T2499 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 729153 T2499 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 729153 T2499 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 729154 T2499 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 729154 T2499 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 729154 T2499 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 729154 T2499 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 729157 T2500 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c48a9[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 729158 T2499 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 729159 T2503 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:48726 collection:control_collection shard:shard1
   [junit4]   2> 729159 T2472 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 729160 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 729160 T2503 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 729166 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 729167 T2505 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1570b6c name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 729167 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 729169 T2472 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 729169 T2503 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 729172 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729172 T2472 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 729172 T2503 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 729173 T2503 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 729173 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C8873 name=collection1 org.apache.solr.core.SolrCore@bd479e url=https://127.0.0.1:48726/collection1 node=127.0.0.1:48726_ C8873_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:48726, core=collection1, node_name=127.0.0.1:48726_}
   [junit4]   2> 729173 T2503 C8873 P48726 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:48726/collection1/
   [junit4]   2> 729174 T2503 C8873 P48726 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 729175 T2503 C8873 P48726 oasc.SyncStrategy.syncToMe https://127.0.0.1:48726/collection1/ has no replicas
   [junit4]   2> 729175 T2503 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:48726/collection1/ shard1
   [junit4]   2> 729175 T2503 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 729178 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729181 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729185 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729288 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 729288 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 729333 T2503 oasc.ZkController.register We are https://127.0.0.1:48726/collection1/ and leader is https://127.0.0.1:48726/collection1/
   [junit4]   2> 729333 T2503 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:48726
   [junit4]   2> 729333 T2503 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 729333 T2503 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 729333 T2503 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 729335 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729335 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729335 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729335 T2503 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 729336 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 729337 T2497 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:48726",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48726_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 729339 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729440 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 729440 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 729447 T2472 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 729448 T2472 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 729453 T2472 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 729454 T2472 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:46945
   [junit4]   2> 729456 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 729456 T2472 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 729457 T2472 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003
   [junit4]   2> 729457 T2472 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/'
   [junit4]   2> 729471 T2472 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/solr.xml
   [junit4]   2> 729483 T2472 oasc.CoreContainer.<init> New CoreContainer 1751242
   [junit4]   2> 729484 T2472 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/]
   [junit4]   2> 729484 T2472 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 729485 T2472 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 729485 T2472 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 729485 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 729486 T2472 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 729486 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 729486 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 729487 T2472 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 729487 T2472 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 729489 T2472 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 729489 T2472 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 729489 T2472 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 729490 T2472 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48431/solr
   [junit4]   2> 729490 T2472 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 729491 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 729493 T2516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a3bb73 name:ZooKeeperConnection Watcher:127.0.0.1:48431 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 729494 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 729497 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 729499 T2518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a00bf9 name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 729499 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 729512 T2472 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730516 T2472 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46945_
   [junit4]   2> 730517 T2472 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46945_
   [junit4]   2> 730527 T2519 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 730528 T2519 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 730529 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 730529 T2519 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 730529 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 730530 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 730530 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 730531 T2497 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46945",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46945_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 730531 T2497 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 730531 T2497 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 730533 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 730636 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 730636 T2518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 730636 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 731530 T2519 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 731530 T2519 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 731532 T2519 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 731532 T2519 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 731533 T2519 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 731533 T2519 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/collection1/'
   [junit4]   2> 731534 T2519 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 731535 T2519 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 731535 T2519 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 731573 T2519 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 731592 T2519 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 731593 T2519 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 731599 T2519 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 731709 T2519 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 731713 T2519 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 731715 T2519 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 731722 T2519 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 731725 T2519 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 731726 T2519 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 731727 T2519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 731728 T2519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 731728 T2519 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 731729 T2519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 731729 T2519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 731729 T2519 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 731730 T2519 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 731730 T2519 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty1/
   [junit4]   2> 731730 T2519 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43456
   [junit4]   2> 731731 T2519 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty1
   [junit4]   2> 731731 T2519 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty1/index/
   [junit4]   2> 731731 T2519 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 731732 T2519 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty1/index
   [junit4]   2> 731732 T2519 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=20.5791015625, floorSegmentMB=0.4296875, forceMergeDeletesPctAllowed=29.988170746820686, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19306883707246503
   [junit4]   2> 731733 T2519 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2949e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@101d542),segFN=segments_1,generation=1}
   [junit4]   2> 731733 T2519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 731737 T2519 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 731737 T2519 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 731737 T2519 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 731737 T2519 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 731738 T2519 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 731738 T2519 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 731738 T2519 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 731739 T2519 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 731739 T2519 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 731739 T2519 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 731740 T2519 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 731740 T2519 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 731741 T2519 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 731741 T2519 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 731742 T2519 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 731742 T2519 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 731743 T2519 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 731745 T2519 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 731746 T2519 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 731747 T2519 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 731747 T2519 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=67.328125, floorSegmentMB=2.181640625, forceMergeDeletesPctAllowed=20.852550126679752, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 731748 T2519 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2949e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@101d542),segFN=segments_1,generation=1}
   [junit4]   2> 731748 T2519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 731749 T2519 oass.SolrIndexSearcher.<init> Opening Searcher@1fbbce5[collection1] main
   [junit4]   2> 731749 T2519 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 731750 T2519 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 731750 T2519 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 731750 T2519 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 731751 T2519 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 731751 T2519 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 731751 T2519 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 731752 T2519 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 731752 T2519 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 731752 T2519 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 731756 T2520 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1fbbce5[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 731760 T2519 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 731761 T2523 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:46945 collection:collection1 shard:shard1
   [junit4]   2> 731761 T2472 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 731762 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 731762 T2523 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 731768 T2523 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 731770 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731771 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731771 T2523 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 731772 T2523 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 731771 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731772 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C8874 name=collection1 org.apache.solr.core.SolrCore@882840 url=https://127.0.0.1:46945/collection1 node=127.0.0.1:46945_ C8874_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:46945, core=collection1, node_name=127.0.0.1:46945_}
   [junit4]   2> 731772 T2523 C8874 P46945 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:46945/collection1/
   [junit4]   2> 731773 T2523 C8874 P46945 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 731774 T2523 C8874 P46945 oasc.SyncStrategy.syncToMe https://127.0.0.1:46945/collection1/ has no replicas
   [junit4]   2> 731774 T2523 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:46945/collection1/ shard1
   [junit4]   2> 731774 T2523 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 731776 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731780 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731783 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731887 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 731887 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 731887 T2518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 731932 T2523 oasc.ZkController.register We are https://127.0.0.1:46945/collection1/ and leader is https://127.0.0.1:46945/collection1/
   [junit4]   2> 731932 T2523 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:46945
   [junit4]   2> 731932 T2523 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 731932 T2523 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 731932 T2523 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 731934 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731934 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731934 T2523 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 731934 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731936 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 731936 T2497 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46945",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46945_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 731938 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 732041 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 732041 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 732041 T2518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 732049 T2472 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 732049 T2472 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 732052 T2472 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 732054 T2472 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45199
   [junit4]   2> 732055 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 732055 T2472 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 732056 T2472 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004
   [junit4]   2> 732056 T2472 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/'
   [junit4]   2> 732069 T2472 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/solr.xml
   [junit4]   2> 732078 T2472 oasc.CoreContainer.<init> New CoreContainer 7041775
   [junit4]   2> 732079 T2472 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/]
   [junit4]   2> 732079 T2472 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 732079 T2472 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 732080 T2472 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 732080 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 732080 T2472 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 732081 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 732081 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 732081 T2472 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 732082 T2472 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 732084 T2472 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 732084 T2472 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 732084 T2472 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 732085 T2472 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48431/solr
   [junit4]   2> 732085 T2472 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 732086 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 732087 T2534 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d04f00 name:ZooKeeperConnection Watcher:127.0.0.1:48431 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 732087 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 732089 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 732090 T2536 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1708a7f name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 732091 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 732096 T2472 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 733099 T2472 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45199_
   [junit4]   2> 733100 T2472 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45199_
   [junit4]   2> 733110 T2537 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 733110 T2537 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 733114 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733114 T2537 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 733114 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733115 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733116 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 733117 T2497 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45199",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45199_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 733117 T2497 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 733117 T2497 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 733120 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733222 T2518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 733222 T2536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 733222 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 733222 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 734114 T2537 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 734115 T2537 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 734116 T2537 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 734116 T2537 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 734118 T2537 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 734118 T2537 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/collection1/'
   [junit4]   2> 734119 T2537 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 734120 T2537 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 734121 T2537 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 734184 T2537 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 734195 T2537 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 734196 T2537 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 734202 T2537 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 734306 T2537 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 734310 T2537 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 734311 T2537 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 734320 T2537 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 734322 T2537 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 734323 T2537 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 734324 T2537 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 734324 T2537 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 734325 T2537 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 734325 T2537 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 734326 T2537 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 734326 T2537 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 734326 T2537 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 734326 T2537 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty2/
   [junit4]   2> 734327 T2537 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43456
   [junit4]   2> 734327 T2537 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty2
   [junit4]   2> 734327 T2537 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty2/index/
   [junit4]   2> 734328 T2537 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 734328 T2537 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty2/index
   [junit4]   2> 734328 T2537 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=20.5791015625, floorSegmentMB=0.4296875, forceMergeDeletesPctAllowed=29.988170746820686, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19306883707246503
   [junit4]   2> 734329 T2537 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b9daf6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ac1e0f),segFN=segments_1,generation=1}
   [junit4]   2> 734329 T2537 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 734333 T2537 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 734333 T2537 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 734334 T2537 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 734334 T2537 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 734334 T2537 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 734334 T2537 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 734335 T2537 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 734335 T2537 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 734335 T2537 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 734336 T2537 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 734337 T2537 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 734337 T2537 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 734337 T2537 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 734339 T2537 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 734339 T2537 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 734340 T2537 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 734341 T2537 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 734343 T2537 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 734345 T2537 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 734345 T2537 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 734346 T2537 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=67.328125, floorSegmentMB=2.181640625, forceMergeDeletesPctAllowed=20.852550126679752, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 734346 T2537 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b9daf6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ac1e0f),segFN=segments_1,generation=1}
   [junit4]   2> 734347 T2537 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 734347 T2537 oass.SolrIndexSearcher.<init> Opening Searcher@f8612f[collection1] main
   [junit4]   2> 734348 T2537 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 734350 T2537 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 734351 T2537 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 734351 T2537 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 734351 T2537 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 734352 T2537 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 734352 T2537 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 734353 T2537 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 734353 T2537 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 734353 T2537 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 734357 T2538 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f8612f[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 734363 T2537 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 734364 T2541 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:45199 collection:collection1 shard:shard1
   [junit4]   2> 734364 T2472 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 734365 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 734366 T2541 oasc.ZkController.register We are https://127.0.0.1:45199/collection1/ and leader is https://127.0.0.1:46945/collection1/
   [junit4]   2> 734367 T2541 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:45199
   [junit4]   2> 734367 T2541 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 734367 T2541 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C8875 name=collection1 org.apache.solr.core.SolrCore@36fd69 url=https://127.0.0.1:45199/collection1 node=127.0.0.1:45199_ C8875_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:45199, core=collection1, node_name=127.0.0.1:45199_}
   [junit4]   2> 734367 T2542 C8875 P45199 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 734367 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 734367 T2542 C8875 P45199 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 734368 T2542 C8875 P45199 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is https://127.0.0.1:46945/collection1/ and I am https://127.0.0.1:45199/collection1/
   [junit4]   2> 734368 T2542 C8875 P45199 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 734368 T2542 C8875 P45199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 734369 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 734370 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 734370 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 734370 T2542 C8875 P45199 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to https://127.0.0.1:46945; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A45199_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
   [junit4]   2> 734372 T2497 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 734372 T2497 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45199",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45199_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 734376 T2496 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 734478 T2505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 734478 T2536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 734478 T2518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 734478 T2496 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 734483 T2511 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 734484 T2511 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 734484 T2511 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 734485 T2511 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:45199_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"state":"recovering","base_url":"https://127.0.0.1:45199","core":"collection1","node_name":"127.0.0.1:45199_"}
   [junit4]   2> 734485 T2511 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
   [junit4]   2> 734485 T2511 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:45199_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=2 
   [junit4]   2> 734624 T2472 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 734624 T2472 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 734627 T2472 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 734628 T2472 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58986
   [junit4]   2> 734632 T2472 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 734632 T2472 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 734633 T2472 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-005
   [junit4]   2> 734634 T2472 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-005/'
   [junit4]   2> 734654 T2472 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-005/solr.xml
   [junit4]   2> 734667 T2472 oasc.CoreContainer.<init> New CoreContainer 28737358
   [junit4]   2> 734668 T2472 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-005/]
   [junit4]   2> 734669 T2472 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 734669 T2472 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 734669 T2472 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 734669 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 734670 T2472 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 734670 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 734670 T2472 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 734671 T2472 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 734671 T2472 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 734673 T2472 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 734673 T2472 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 734674 T2472 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 734674 T2472 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48431/solr
   [junit4]   2> 734674 T2472 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 734675 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 734678 T2554 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@986927 name:ZooKeeperConnection Watcher:127.0.0.1:48431 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 734678 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 734681 T2472 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 734682 T2556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c4c46e name:ZooKeeperConnection Watcher:127.0.0.1:48431/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 734682 T2472 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 734690 T2472 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 735694 T2472 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.

[...truncated too long message...]

est-B7F016F133CCE3E5-001/tempDir-001/jetty4/index
   [junit4]   2> 786564 T2577 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty4;done=false>>]
   [junit4]   2> 786565 T2577 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001/tempDir-001/jetty4
   [junit4]   2> 786565 T2577 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:313)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:297)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:190)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 786566 T2577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 786566 T2577 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 786566 T2577 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 786567 T2577 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.SyncSliceTest-B7F016F133CCE3E5-001
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=et, timezone=Europe/Prague
   [junit4]   2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.8.0_05 (32-bit)/cpus=8,threads=1,free=136880584,total=305618944
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsLockFactoryTest, SolrRequestParserTest, TestHighFrequencyDictionaryFactory, TestFiltering, BasicDistributedZk2Test, StandardRequestHandlerTest, TestCollationField, JSONWriterTest, TestAnalyzedSuggestions, ReplicationFactorTest, TestCoreDiscovery, TestRestManager, TestInfoStreamLogging, TestManagedResourceStorage, ResourceLoaderTest, DisMaxRequestHandlerTest, TestXIncludeConfig, SuggesterTSTTest, TestIndexingPerformance, TestSchemaNameResource, TestCloudManagedSchema, DirectUpdateHandlerTest, TestRandomFaceting, DateFieldTest, HighlighterConfigTest, TestClassNameShortening, TestArbitraryIndexDir, TestCollapseQParserPlugin, XsltUpdateRequestHandlerTest, SortByFunctionTest, DistributedSuggestComponentTest, TestIBSimilarityFactory, EnumFieldTest, TestReload, TestMaxScoreQueryParser, DistributedExpandComponentTest, TestRangeQuery, TestSolr4Spatial, SolrCoreCheckLockOnStartupTest, FieldFacetExtrasTest, ConnectionManagerTest, ShardRoutingCustomTest, TestManagedSynonymFilterFactory, AddBlockUpdateTest, DirectSolrConnectionTest, OpenExchangeRatesOrgProviderTest, HardAutoCommitTest, TestPartialUpdateDeduplication, TestSolrCoreProperties, TestDistributedGrouping, TestQueryTypes, TestLazyCores, PingRequestHandlerTest, AlternateDirectoryTest, TestZkChroot, TestFuzzyAnalyzedSuggestions, TestCSVLoader, TestFieldResource, QueryFacetTest, DocValuesMissingTest, HdfsUnloadDistributedZkTest, TestOrdValues, SolrPluginUtilsTest, TestSolrJ, TestOmitPositions, MinimalSchemaTest, TestSolrQueryParserDefaultOperatorResource, SuggestComponentTest, DeleteReplicaTest, PathHierarchyTokenizerFactoryTest, TestFreeTextSuggestions, TestImplicitCoreProperties, TestWordDelimiterFilterFactory, FileUtilsTest, DirectSolrSpellCheckerTest, ZkNodePropsTest, DateMathParserTest, TestSerializedLuceneMatchVersion, TestRecovery, HighlighterTest, NoFacetTest, TestRTGBase, CurrencyFieldXmlFileTest, RequiredFieldsTest, MergeStrategyTest, RequestHandlersTest, TestHashPartitioner, DistanceFunctionTest, DistribCursorPagingTest, FunctionTest, TestLRUCache, TestStressReorder, TestCSVResponseWriter, NoCacheHeaderTest, InfoHandlerTest, BasicZkTest, TestDistribDocBasedVersion, SyncSliceTest]
   [junit4] Completed on J1 in 60.94s, 1 test, 1 error <<< FAILURES!

[...truncated 624 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 error, 36 ignored (15 assumptions)

Total time: 67 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message