lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b96) - Build # 6483 - Failure!
Date Tue, 09 Jul 2013 13:03:32 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6483/
Java: 32bit/jdk1.8.0-ea-b96 -server -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.schema.TestCloudManagedSchemaAddField.testDistribSearch

Error Message:
Max retry count 20 exceeded after 434 ms.  QUERY FAILED: xpath=/response/lst[@name='responseHeader']/int[@name='status'][.='0']  request=/schema/fields/newfield3?wt=xml  response=<?xml version="1.0" encoding="UTF-8"?> <response>  <lst name="responseHeader">   <int name="status">404</int>   <int name="QTime">1</int> </lst> <lst name="error">   <str name="msg">Field 'newfield3' not found.</str>   <int name="code">404</int> </lst> </response> 

Stack Trace:
java.lang.AssertionError: Max retry count 20 exceeded after 434 ms.  QUERY FAILED: xpath=/response/lst[@name='responseHeader']/int[@name='status'][.='0']  request=/schema/fields/newfield3?wt=xml  response=<?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">404</int>
  <int name="QTime">1</int>
</lst>
<lst name="error">
  <str name="msg">Field 'newfield3' not found.</str>
  <int name="code">404</int>
</lst>
</response>

	at __randomizedtesting.SeedInfo.seed([141EDF8B23989943:95F8519354C7F97F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.schema.TestCloudManagedSchemaAddField.doTest(TestCloudManagedSchemaAddField.java:119)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:491)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9336 lines...]
   [junit4] Suite: org.apache.solr.schema.TestCloudManagedSchemaAddField
   [junit4]   2> 36562 T167 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 36566 T167 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestCloudManagedSchemaAddField-1373374083241
   [junit4]   2> 36570 T167 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 36572 T168 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 36672 T167 oasc.ZkTestServer.run start zk server on port:54012
   [junit4]   2> 36730 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 36826 T174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e7e8e name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 36826 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 36828 T167 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 36862 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 36864 T176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7db266 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 36864 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 36867 T167 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 36872 T167 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 36875 T167 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 36878 T167 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 36882 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog-mutable-managed-schema.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 36883 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 36888 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 36889 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 36893 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 36894 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 36897 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 36897 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 36900 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 36901 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 36904 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 36905 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 36908 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 36909 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 36912 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 36912 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 36915 T167 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 36916 T167 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 37005 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 37008 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59178
   [junit4]   2> 37009 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 37009 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 37010 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595
   [junit4]   2> 37010 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/'
   [junit4]   2> 37030 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/solr.xml
   [junit4]   2> 37038 T167 oasc.CoreContainer.<init> New CoreContainer 28258397
   [junit4]   2> 37039 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/]
   [junit4]   2> 37039 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 37040 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 37040 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 37040 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 37040 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 37041 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 37041 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 37041 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 37042 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 37042 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 37046 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 37047 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 37055 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 37056 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37078 T187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a56b7 name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37078 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37081 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 37086 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37089 T189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1632cf2 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37089 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37095 T167 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 37101 T167 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 37108 T167 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 37109 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59178_
   [junit4]   2> 37112 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59178_
   [junit4]   2> 37116 T167 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 37128 T167 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 37130 T167 oasc.Overseer.start Overseer (id=90005443926491139-127.0.0.1:59178_-n_0000000000) starting
   [junit4]   2> 37142 T167 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 37152 T191 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 37153 T167 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 37156 T167 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 37159 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 37172 T190 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 37175 T192 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 37176 T192 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 37178 T192 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 38679 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 38682 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59178_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59178"}
   [junit4]   2> 38682 T190 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
   [junit4]   2> 38689 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 38703 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 39178 T192 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 39179 T192 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/collection1
   [junit4]   2> 39179 T192 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 39180 T192 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 39180 T192 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 39181 T192 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/collection1/'
   [junit4]   2> 39182 T192 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/collection1/lib/README' to classloader
   [junit4]   2> 39182 T192 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/collection1/lib/classes/' to classloader
   [junit4]   2> 39205 T192 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 39214 T192 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 39220 T192 oass.ManagedIndexSchemaFactory.create The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 39223 T192 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 39229 T192 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 39344 T192 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 39349 T192 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 39350 T192 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 39368 T192 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 39371 T192 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 39373 T192 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 39375 T192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 39375 T192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 39375 T192 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 39376 T192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 39377 T192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 39405 T192 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Created and persisted managed schema znode at /configs/conf1/managed-schema
   [junit4]   2> 39407 T192 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml.bak
   [junit4]   2> 39413 T192 oass.ManagedIndexSchemaFactory.zkUgradeToManagedSchema After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/conf1/schema.xml to /configs/conf1/schema.xml.bak
   [junit4]   2> 39413 T192 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 39413 T192 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-controljetty-1373374083595/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/
   [junit4]   2> 39413 T192 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 39414 T192 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data
   [junit4]   2> 39415 T192 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/index/
   [junit4]   2> 39415 T192 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 39415 T192 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/index
   [junit4]   2> 39417 T192 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13890a1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1765401),segFN=segments_1,generation=1}
   [junit4]   2> 39417 T192 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 39419 T192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 39419 T192 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 39420 T192 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 39420 T192 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 39420 T192 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 39421 T192 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 39421 T192 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 39423 T192 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 39426 T192 oass.SolrIndexSearcher.<init> Opening Searcher@1dce02 main
   [junit4]   2> 39427 T192 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 39427 T192 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 39428 T192 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 39433 T193 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dce02 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 39434 T192 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 39434 T192 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59178 collection:control_collection shard:shard1
   [junit4]   2> 39438 T192 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 39443 T192 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 39451 T192 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 39454 T192 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 39455 T192 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 39456 T192 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59178/collection1/
   [junit4]   2> 39456 T192 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 39456 T192 oasc.SyncStrategy.syncToMe http://127.0.0.1:59178/collection1/ has no replicas
   [junit4]   2> 39456 T192 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59178/collection1/ shard1
   [junit4]   2> 39457 T192 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 40208 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 40217 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 40266 T192 oasc.ZkController.register We are http://127.0.0.1:59178/collection1/ and leader is http://127.0.0.1:59178/collection1/
   [junit4]   2> 40266 T192 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59178
   [junit4]   2> 40267 T192 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 40267 T192 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 40267 T192 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 40269 T192 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 40270 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 40271 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 40272 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 40277 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 40280 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 40282 T196 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df3ac5 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 40282 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 40285 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 40289 T167 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 40363 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 40365 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60697
   [junit4]   2> 40365 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 40365 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 40366 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964
   [junit4]   2> 40366 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/'
   [junit4]   2> 40377 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/solr.xml
   [junit4]   2> 40387 T167 oasc.CoreContainer.<init> New CoreContainer 19210768
   [junit4]   2> 40388 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/]
   [junit4]   2> 40388 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 40389 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 40389 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 40389 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 40390 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 40390 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 40390 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 40390 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 40391 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 40391 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 40395 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 40395 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 40396 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 40396 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 40398 T207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8344ff name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 40398 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 40400 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 40404 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 40410 T209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c881d1 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 40410 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 40415 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 41418 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60697_
   [junit4]   2> 41419 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60697_
   [junit4]   2> 41423 T196 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 41423 T209 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 41423 T189 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 41423 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41429 T210 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 41430 T210 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 41431 T210 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 41723 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 41723 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59178_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59178"}
   [junit4]   2> 41728 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60697_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60697"}
   [junit4]   2> 41728 T190 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 4
   [junit4]   2> 41728 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 41733 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41733 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41733 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 42431 T210 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 42431 T210 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/collection1
   [junit4]   2> 42432 T210 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 42433 T210 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 42433 T210 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 42434 T210 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/collection1/'
   [junit4]   2> 42435 T210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/collection1/lib/README' to classloader
   [junit4]   2> 42435 T210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/collection1/lib/classes/' to classloader
   [junit4]   2> 42457 T210 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 42469 T210 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 42471 T210 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 42483 T210 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 42645 T210 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 42650 T210 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 42651 T210 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 42656 T210 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 42660 T210 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 42674 T210 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 42675 T210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 42676 T210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 42676 T210 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 42678 T210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 42678 T210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 42684 T210 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 42684 T210 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty1-1373374086964/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/
   [junit4]   2> 42685 T210 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 42686 T210 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1
   [junit4]   2> 42686 T210 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/index/
   [junit4]   2> 42686 T210 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 42686 T210 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/index
   [junit4]   2> 42688 T210 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@51834e),segFN=segments_1,generation=1}
   [junit4]   2> 42688 T210 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 42691 T210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 42691 T210 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 42692 T210 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 42692 T210 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 42692 T210 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 42693 T210 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 42693 T210 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 42696 T210 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 42697 T210 oass.SolrIndexSearcher.<init> Opening Searcher@5e50f0 main
   [junit4]   2> 42697 T210 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 42697 T210 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 42698 T210 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 42708 T211 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5e50f0 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 42709 T210 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 42709 T210 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60697 collection:collection1 shard:shard1
   [junit4]   2> 42709 T210 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 42713 T210 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 42750 T210 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 42754 T210 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 42754 T210 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 42754 T210 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60697/collection1/
   [junit4]   2> 42755 T210 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 42755 T210 oasc.SyncStrategy.syncToMe http://127.0.0.1:60697/collection1/ has no replicas
   [junit4]   2> 42755 T210 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60697/collection1/ shard1
   [junit4]   2> 42755 T210 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 43239 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 43248 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 43248 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 43248 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 43263 T210 oasc.ZkController.register We are http://127.0.0.1:60697/collection1/ and leader is http://127.0.0.1:60697/collection1/
   [junit4]   2> 43263 T210 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60697
   [junit4]   2> 43263 T210 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 43264 T210 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 43264 T210 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 43266 T210 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 43267 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 43267 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 43268 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 43354 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 43358 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53821
   [junit4]   2> 43359 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 43359 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 43360 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947
   [junit4]   2> 43360 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/'
   [junit4]   2> 43378 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/solr.xml
   [junit4]   2> 43387 T167 oasc.CoreContainer.<init> New CoreContainer 20382611
   [junit4]   2> 43388 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/]
   [junit4]   2> 43389 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 43389 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 43389 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 43389 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 43390 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 43390 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 43390 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 43391 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 43391 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 43391 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 43395 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 43396 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 43396 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 43397 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 43399 T223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d3e35a name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 43399 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 43401 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 43404 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 43406 T225 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@136df60 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 43406 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 43410 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 44414 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53821_
   [junit4]   2> 44436 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53821_
   [junit4]   2> 44457 T196 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 44458 T209 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 44457 T225 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 44458 T189 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 44458 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44458 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44458 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44478 T226 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 44479 T226 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 44480 T226 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 44755 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44756 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60697_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60697"}
   [junit4]   2> 44761 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53821_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53821"}
   [junit4]   2> 44761 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 44761 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard4
   [junit4]   2> 44767 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44767 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44767 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44767 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 45480 T226 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 45480 T226 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/collection1
   [junit4]   2> 45481 T226 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 45482 T226 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 45482 T226 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 45484 T226 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/collection1/'
   [junit4]   2> 45485 T226 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/collection1/lib/README' to classloader
   [junit4]   2> 45485 T226 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/collection1/lib/classes/' to classloader
   [junit4]   2> 45519 T226 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 45540 T226 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 45545 T226 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 45555 T226 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 45764 T226 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 45768 T226 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 45769 T226 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 45773 T226 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 45777 T226 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 45792 T226 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 45793 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 45793 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 45794 T226 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 45796 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 45797 T226 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 45801 T226 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 45802 T226 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty2-1373374089947/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/
   [junit4]   2> 45802 T226 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 45803 T226 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2
   [junit4]   2> 45803 T226 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/index/
   [junit4]   2> 45803 T226 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 45804 T226 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/index
   [junit4]   2> 45806 T226 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@900e3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d535a0),segFN=segments_1,generation=1}
   [junit4]   2> 45806 T226 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 45809 T226 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 45810 T226 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 45811 T226 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 45811 T226 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 45811 T226 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 45812 T226 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 45812 T226 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 45815 T226 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 45817 T226 oass.SolrIndexSearcher.<init> Opening Searcher@1276057 main
   [junit4]   2> 45819 T226 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 45819 T226 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 45820 T226 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 45825 T227 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1276057 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 45828 T226 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 45828 T226 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53821 collection:collection1 shard:shard4
   [junit4]   2> 45829 T226 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 45835 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard4/election
   [junit4]   2> 45841 T226 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard4
   [junit4]   2> 45844 T226 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 45844 T226 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 45845 T226 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53821/collection1/
   [junit4]   2> 45845 T226 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 45845 T226 oasc.SyncStrategy.syncToMe http://127.0.0.1:53821/collection1/ has no replicas
   [junit4]   2> 45845 T226 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53821/collection1/ shard4
   [junit4]   2> 45846 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard4
   [junit4]   2> 46274 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 46284 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 46284 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 46284 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 46284 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 46327 T226 oasc.ZkController.register We are http://127.0.0.1:53821/collection1/ and leader is http://127.0.0.1:53821/collection1/
   [junit4]   2> 46327 T226 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53821
   [junit4]   2> 46327 T226 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 46328 T226 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 46328 T226 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 46331 T226 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 46332 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 46333 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 46333 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 46457 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 46471 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47013
   [junit4]   2> 46472 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 46472 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 46473 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012
   [junit4]   2> 46473 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/'
   [junit4]   2> 46487 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/solr.xml
   [junit4]   2> 46502 T167 oasc.CoreContainer.<init> New CoreContainer 18286318
   [junit4]   2> 46502 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/]
   [junit4]   2> 46503 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 46503 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 46503 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 46504 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 46504 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 46504 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 46504 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 46505 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 46505 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 46505 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 46509 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 46509 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 46510 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 46511 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46513 T239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de0a72 name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46513 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46515 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 46519 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46520 T241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7889c6 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46521 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46525 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 47528 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47013_
   [junit4]   2> 47529 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47013_
   [junit4]   2> 47532 T189 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 47532 T225 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 47532 T196 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 47532 T209 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 47532 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47532 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47532 T241 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 47532 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47532 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47539 T242 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 47540 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 47541 T242 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 47794 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 47795 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53821_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53821"}
   [junit4]   2> 47892 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47013_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47013"}
   [junit4]   2> 47892 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 47893 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 47898 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47898 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47898 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47898 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 47898 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 48542 T242 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 48542 T242 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/collection1
   [junit4]   2> 48542 T242 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 48543 T242 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 48543 T242 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 48545 T242 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/collection1/'
   [junit4]   2> 48545 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/collection1/lib/README' to classloader
   [junit4]   2> 48546 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/collection1/lib/classes/' to classloader
   [junit4]   2> 48567 T242 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 48580 T242 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 48582 T242 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 48592 T242 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 48778 T242 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 48783 T242 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 48785 T242 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 48790 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 48793 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 48806 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 48808 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 48808 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 48808 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 48810 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 48810 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 48815 T242 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 48815 T242 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty3-1373374093012/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/
   [junit4]   2> 48815 T242 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 48816 T242 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3
   [junit4]   2> 48816 T242 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/index/
   [junit4]   2> 48816 T242 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 48817 T242 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/index
   [junit4]   2> 48819 T242 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@124bc24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@154d388),segFN=segments_1,generation=1}
   [junit4]   2> 48819 T242 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 48822 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 48823 T242 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 48823 T242 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 48824 T242 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 48824 T242 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 48825 T242 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 48825 T242 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 48828 T242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 48829 T242 oass.SolrIndexSearcher.<init> Opening Searcher@ed9175 main
   [junit4]   2> 48830 T242 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 48830 T242 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 48831 T242 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 48836 T243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ed9175 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 48841 T242 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 48841 T242 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47013 collection:collection1 shard:shard2
   [junit4]   2> 48841 T242 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 48845 T242 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 48854 T242 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 48858 T242 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 48858 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 48858 T242 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47013/collection1/
   [junit4]   2> 48858 T242 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 48859 T242 oasc.SyncStrategy.syncToMe http://127.0.0.1:47013/collection1/ has no replicas
   [junit4]   2> 48859 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47013/collection1/ shard2
   [junit4]   2> 48859 T242 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 49407 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 49417 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 49417 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 49417 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 49417 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 49417 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 49469 T242 oasc.ZkController.register We are http://127.0.0.1:47013/collection1/ and leader is http://127.0.0.1:47013/collection1/
   [junit4]   2> 49469 T242 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47013
   [junit4]   2> 49469 T242 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 49469 T242 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 49470 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 49472 T242 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 49473 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 49474 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 49474 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 49590 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 49592 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54899
   [junit4]   2> 49593 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 49593 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 49594 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153
   [junit4]   2> 49594 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/'
   [junit4]   2> 49612 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/solr.xml
   [junit4]   2> 49631 T167 oasc.CoreContainer.<init> New CoreContainer 4421876
   [junit4]   2> 49632 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/]
   [junit4]   2> 49633 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 49633 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 49634 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 49634 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 49635 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 49635 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 49635 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 49636 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 49636 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 49637 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 49642 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 49643 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 49644 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 49645 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 49647 T255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17a14fa name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 49647 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 49649 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 49655 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 49658 T257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d47481 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 49658 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 49663 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 50672 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54899_
   [junit4]   2> 50674 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54899_
   [junit4]   2> 50676 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 50677 T209 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 50677 T225 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 50677 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50677 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50678 T196 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 50678 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50679 T257 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 50678 T189 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 50679 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50683 T241 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 50686 T258 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 50686 T258 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 50688 T258 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 50926 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 50927 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47013_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47013"}
   [junit4]   2> 50931 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node4",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54899_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54899"}
   [junit4]   2> 50932 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 50932 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 50945 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50946 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50946 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50945 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50946 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 50946 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 51688 T258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 51689 T258 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/collection1
   [junit4]   2> 51689 T258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 51690 T258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 51690 T258 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 51691 T258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/collection1/'
   [junit4]   2> 51692 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/collection1/lib/README' to classloader
   [junit4]   2> 51692 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/collection1/lib/classes/' to classloader
   [junit4]   2> 51710 T258 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 51723 T258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 51725 T258 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 51734 T258 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 51935 T258 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 51940 T258 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 51942 T258 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 51946 T258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 51950 T258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 51997 T258 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 51999 T258 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 52000 T258 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 52000 T258 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 52002 T258 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 52003 T258 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 52010 T258 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 52010 T258 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty4-1373374096153/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/
   [junit4]   2> 52011 T258 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 52012 T258 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4
   [junit4]   2> 52012 T258 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/index/
   [junit4]   2> 52012 T258 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/index' doesn't exist. Creating new index...
   [junit4]   2> 52012 T258 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/index
   [junit4]   2> 52014 T258 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15b2994 lockFactory=org.apache.lucene.store.NativeFSLockFactory@147d3d7),segFN=segments_1,generation=1}
   [junit4]   2> 52015 T258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52018 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 52019 T258 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 52020 T258 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 52020 T258 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 52021 T258 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 52022 T258 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 52022 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 52025 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 52027 T258 oass.SolrIndexSearcher.<init> Opening Searcher@c8f485 main
   [junit4]   2> 52027 T258 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 52028 T258 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 52028 T258 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 52035 T259 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c8f485 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 52037 T258 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 52038 T258 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54899 collection:collection1 shard:shard3
   [junit4]   2> 52038 T258 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 52046 T258 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 52055 T258 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 52057 T258 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 52057 T258 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 52058 T258 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54899/collection1/
   [junit4]   2> 52058 T258 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 52058 T258 oasc.SyncStrategy.syncToMe http://127.0.0.1:54899/collection1/ has no replicas
   [junit4]   2> 52058 T258 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54899/collection1/ shard3
   [junit4]   2> 52059 T258 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 52456 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52465 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 52465 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 52466 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 52465 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 52465 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 52466 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 52516 T258 oasc.ZkController.register We are http://127.0.0.1:54899/collection1/ and leader is http://127.0.0.1:54899/collection1/
   [junit4]   2> 52516 T258 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54899
   [junit4]   2> 52516 T258 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 52516 T258 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 52516 T258 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 52518 T258 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52520 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 52520 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 52521 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 52633 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 52635 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38332
   [junit4]   2> 52636 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 52637 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 52637 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198
   [junit4]   2> 52638 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/'
   [junit4]   2> 52655 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/solr.xml
   [junit4]   2> 52670 T167 oasc.CoreContainer.<init> New CoreContainer 20888152
   [junit4]   2> 52670 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/]
   [junit4]   2> 52671 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 52671 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 52672 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 52672 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 52672 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 52673 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 52673 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 52673 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 52674 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 52674 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 52681 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 52682 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 52682 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 52692 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52695 T271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@103ff97 name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52695 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52698 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 52704 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52707 T273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d6a17 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52707 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52711 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 53714 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38332_
   [junit4]   2> 53716 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38332_
   [junit4]   2> 53719 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 53719 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 53723 T196 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53723 T189 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53723 T209 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53724 T225 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53724 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53723 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53723 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53724 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53725 T257 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53726 T273 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53727 T241 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 53734 T274 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 53734 T274 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 53736 T274 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 53974 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53975 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node4",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54899_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54899"}
   [junit4]   2> 53979 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node5",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38332_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38332"}
   [junit4]   2> 53980 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 53980 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 53986 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53986 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53986 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53986 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53986 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53986 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 53986 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 54739 T274 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 54740 T274 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/collection1
   [junit4]   2> 54740 T274 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 54741 T274 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 54741 T274 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 54743 T274 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/collection1/'
   [junit4]   2> 54743 T274 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/collection1/lib/README' to classloader
   [junit4]   2> 54744 T274 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/collection1/lib/classes/' to classloader
   [junit4]   2> 54773 T274 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 54786 T274 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 54788 T274 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 54795 T274 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 54968 T274 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 54974 T274 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 54975 T274 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 54980 T274 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 54983 T274 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 54994 T274 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 54995 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 54996 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 54996 T274 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 54998 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 54998 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 55001 T274 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 55002 T274 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty5-1373374099198/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/
   [junit4]   2> 55002 T274 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 55003 T274 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5
   [junit4]   2> 55003 T274 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/index/
   [junit4]   2> 55003 T274 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/index' doesn't exist. Creating new index...
   [junit4]   2> 55003 T274 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/index
   [junit4]   2> 55004 T274 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1ddb50d lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bbf01),segFN=segments_1,generation=1}
   [junit4]   2> 55005 T274 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 55007 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 55007 T274 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 55008 T274 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 55008 T274 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 55008 T274 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 55009 T274 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 55009 T274 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 55011 T274 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 55012 T274 oass.SolrIndexSearcher.<init> Opening Searcher@a558a2 main
   [junit4]   2> 55012 T274 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 55012 T274 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 55013 T274 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 55017 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a558a2 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 55019 T274 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 55019 T274 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38332 collection:collection1 shard:shard1
   [junit4]   2> 55020 T274 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 55030 T274 oasc.ZkController.register We are http://127.0.0.1:38332/collection1/ and leader is http://127.0.0.1:60697/collection1/
   [junit4]   2> 55031 T274 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38332
   [junit4]   2> 55031 T274 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 55031 T274 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C188 name=collection1 org.apache.solr.core.SolrCore@6a59e7 url=http://127.0.0.1:38332/collection1 node=127.0.0.1:38332_ C188_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:38332_, base_url=http://127.0.0.1:38332}
   [junit4]   2> 55035 T276 C188 P38332 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 55035 T274 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55035 T276 C188 P38332 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 55036 T276 C188 P38332 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 55036 T276 C188 P38332 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 55036 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 55037 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 55037 T276 C188 P38332 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 55037 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 55046 T205 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node5, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 55119 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 55127 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48632
   [junit4]   2> 55128 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 55129 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 55129 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715
   [junit4]   2> 55129 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/'
   [junit4]   2> 55145 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/solr.xml
   [junit4]   2> 55153 T167 oasc.CoreContainer.<init> New CoreContainer 14118405
   [junit4]   2> 55153 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/]
   [junit4]   2> 55154 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 55154 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 55154 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 55155 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 55155 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 55155 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 55155 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 55156 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 55156 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 55156 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 55166 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 55167 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 55167 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 55168 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55171 T288 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f86c4c name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55171 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55173 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 55178 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55180 T290 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1539194 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55180 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55184 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 55495 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55496 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node5",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38332_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38332"}
   [junit4]   2> 55501 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55502 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55502 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55501 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55501 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55501 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55502 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 55502 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 56046 T205 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 56047 T205 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node5&state=recovering&nodeName=127.0.0.1:38332_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003 
   [junit4]   2> 56188 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48632_
   [junit4]   2> 56189 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48632_
   [junit4]   2> 56192 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 56192 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 56193 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 56194 T209 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56194 T189 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56194 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 56194 T196 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56194 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 56194 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 56194 T225 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56195 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 56195 T290 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56196 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 56196 T241 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56196 T273 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56198 T257 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 56203 T291 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 56203 T291 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 56205 T291 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 57011 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 57012 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node6",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48632_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48632"}
   [junit4]   2> 57012 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 57012 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard4
   [junit4]   2> 57017 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57018 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 57205 T291 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 57205 T291 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/collection1
   [junit4]   2> 57206 T291 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 57206 T291 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 57207 T291 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 57209 T291 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/collection1/'
   [junit4]   2> 57209 T291 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/collection1/lib/README' to classloader
   [junit4]   2> 57210 T291 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/collection1/lib/classes/' to classloader
   [junit4]   2> 57227 T291 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 57237 T291 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 57239 T291 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 57244 T291 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 57430 T291 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 57434 T291 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 57435 T291 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 57439 T291 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 57442 T291 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 57455 T291 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 57456 T291 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 57456 T291 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 57456 T291 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 57458 T291 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 57458 T291 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 57462 T291 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 57462 T291 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty6-1373374101715/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/
   [junit4]   2> 57462 T291 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 57463 T291 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6
   [junit4]   2> 57463 T291 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/index/
   [junit4]   2> 57463 T291 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/index' doesn't exist. Creating new index...
   [junit4]   2> 57464 T291 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/index
   [junit4]   2> 57466 T291 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1241cb6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@15f9127),segFN=segments_1,generation=1}
   [junit4]   2> 57466 T291 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 57469 T291 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 57469 T291 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 57470 T291 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 57470 T291 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 57470 T291 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 57471 T291 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 57471 T291 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 57474 T291 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 57475 T291 oass.SolrIndexSearcher.<init> Opening Searcher@1d47161 main
   [junit4]   2> 57476 T291 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 57476 T291 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 57476 T291 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 57488 T292 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d47161 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 57489 T291 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 57489 T291 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48632 collection:collection1 shard:shard4
   [junit4]   2> 57489 T291 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 57503 T291 oasc.ZkController.register We are http://127.0.0.1:48632/collection1/ and leader is http://127.0.0.1:53821/collection1/
   [junit4]   2> 57503 T291 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48632
   [junit4]   2> 57503 T291 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 57503 T291 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C189 name=collection1 org.apache.solr.core.SolrCore@bf0d24 url=http://127.0.0.1:48632/collection1 node=127.0.0.1:48632_ C189_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:48632_, base_url=http://127.0.0.1:48632}
   [junit4]   2> 57504 T293 C189 P48632 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 57504 T291 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 57505 T293 C189 P48632 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 57505 T293 C189 P48632 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 57505 T293 C189 P48632 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 57506 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 57506 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 57507 T293 C189 P48632 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 57507 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 57517 T219 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 57599 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 57601 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56419
   [junit4]   2> 57602 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 57603 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 57603 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186
   [junit4]   2> 57603 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/'
   [junit4]   2> 57617 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/solr.xml
   [junit4]   2> 57638 T167 oasc.CoreContainer.<init> New CoreContainer 13873096
   [junit4]   2> 57638 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/]
   [junit4]   2> 57639 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 57639 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 57639 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 57640 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 57640 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 57640 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 57640 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 57641 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 57641 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 57641 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 57646 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 57646 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 57647 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 57647 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 57649 T305 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ba7021 name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 57650 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 57652 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 57656 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 57659 T307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e4ba58 name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 57659 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 57663 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C190 name=collection1 org.apache.solr.core.SolrCore@6a59e7 url=http://127.0.0.1:38332/collection1 node=127.0.0.1:38332_ C190_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:38332_, base_url=http://127.0.0.1:38332}
   [junit4]   2> 58049 T276 C190 P38332 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:60697/collection1/ core=collection1 - recoveringAfterStartup=true
   [junit4]   2> 58054 T276 C190 P38332 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 58061 T276 C190 P38332 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38332 START replicas=[http://127.0.0.1:60697/collection1/] nUpdates=100
   [junit4]   2> 58062 T276 C190 P38332 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
   [junit4]   2> 58065 T276 C190 P38332 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
   [junit4]   2> 58066 T276 C190 P38332 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
   [junit4]   2> 58066 T276 C190 P38332 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
   [junit4]   2> 58066 T276 C190 P38332 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 58066 T276 C190 P38332 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:60697/collection1/. core=collection1
   [junit4]   2> 58067 T276 C190 P38332 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> ASYNC  NEW_CORE C191 name=collection1 org.apache.solr.core.SolrCore@6fc219 url=http://127.0.0.1:60697/collection1 node=127.0.0.1:60697_ C191_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60697_, base_url=http://127.0.0.1:60697, leader=true}
   [junit4]   2> 58069 T203 C191 P60697 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
   [junit4]   2> 58080 T202 C191 P60697 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 58082 T202 C191 P60697 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@51834e),segFN=segments_1,generation=1}
   [junit4]   2> 58082 T202 C191 P60697 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 58083 T202 C191 P60697 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 58083 T202 C191 P60697 oass.SolrIndexSearcher.<init> Opening Searcher@61cba realtime
   [junit4]   2> 58083 T202 C191 P60697 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 58084 T202 C191 P60697 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
   [junit4]   2> 58112 T276 C190 P38332 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 58112 T276 C190 P38332 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
   [junit4]   2> 58117 T276 C190 P38332 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 58360 T205 C191 P60697 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 58360 T205 C191 P60697 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=240 
   [junit4]   2> 58361 T276 C190 P38332 oasc.RecoveryStrategy.replay No replay needed. core=collection1
   [junit4]   2> 58361 T276 C190 P38332 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
   [junit4]   2> 58361 T276 C190 P38332 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 58362 T276 C190 P38332 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 58363 T276 C190 P38332 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
   [junit4]   2> 58527 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58528 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node6",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48632_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48632"}
   [junit4]   2> 58531 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node5",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38332_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38332"}
   [junit4]   2> 58535 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58535 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58535 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58536 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58536 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58536 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58536 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58536 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58535 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58667 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56419_
   [junit4]   2> 58668 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56419_
   [junit4]   2> 58670 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58671 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58671 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58671 T290 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58671 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 58672 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 58672 T209 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58672 T225 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58672 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 58673 T189 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58673 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 58672 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 58673 T196 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58674 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 58676 T273 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58677 T307 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58677 T257 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58677 T241 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 58685 T309 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 58685 T309 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 58690 T309 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 59517 T219 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
   [junit4]   2> 59518 T219 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node6&state=recovering&nodeName=127.0.0.1:48632_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001 
   [junit4]   2> 60043 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 60044 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node7",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56419_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56419"}
   [junit4]   2> 60045 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 60045 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 60049 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60049 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60049 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60050 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60049 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60049 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60049 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60050 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60050 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 60691 T309 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 60691 T309 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/collection1
   [junit4]   2> 60691 T309 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 60692 T309 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 60692 T309 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 60694 T309 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/collection1/'
   [junit4]   2> 60694 T309 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/collection1/lib/README' to classloader
   [junit4]   2> 60694 T309 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/collection1/lib/classes/' to classloader
   [junit4]   2> 60716 T309 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 60739 T309 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 60740 T309 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 60748 T309 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 60937 T309 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 60942 T309 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 60943 T309 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 60946 T309 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 60949 T309 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 60960 T309 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 60961 T309 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 60961 T309 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 60962 T309 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 60963 T309 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 60963 T309 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 60968 T309 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 60968 T309 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty7-1373374104186/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/
   [junit4]   2> 60968 T309 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 60969 T309 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7
   [junit4]   2> 60969 T309 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/index/
   [junit4]   2> 60969 T309 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/index' doesn't exist. Creating new index...
   [junit4]   2> 60970 T309 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/index
   [junit4]   2> 60971 T309 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@510682 lockFactory=org.apache.lucene.store.NativeFSLockFactory@af8ae6),segFN=segments_1,generation=1}
   [junit4]   2> 60971 T309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 60974 T309 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 60975 T309 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 60975 T309 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 60975 T309 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 60976 T309 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 60976 T309 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 60976 T309 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 60978 T309 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 60979 T309 oass.SolrIndexSearcher.<init> Opening Searcher@67f068 main
   [junit4]   2> 60980 T309 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 60980 T309 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 60981 T309 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 60986 T310 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67f068 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 60988 T309 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 60988 T309 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56419 collection:collection1 shard:shard2
   [junit4]   2> 60988 T309 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 60997 T309 oasc.ZkController.register We are http://127.0.0.1:56419/collection1/ and leader is http://127.0.0.1:47013/collection1/
   [junit4]   2> 60998 T309 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56419
   [junit4]   2> 60998 T309 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 60998 T309 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C192 name=collection1 org.apache.solr.core.SolrCore@177b244 url=http://127.0.0.1:56419/collection1 node=127.0.0.1:56419_ C192_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:56419_, base_url=http://127.0.0.1:56419}
   [junit4]   2> 60998 T311 C192 P56419 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 60999 T311 C192 P56419 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 60999 T311 C192 P56419 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 60999 T309 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 60999 T311 C192 P56419 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 61000 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 61001 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 61000 T311 C192 P56419 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 61002 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 61009 T234 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node7, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 61096 T167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 61098 T167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40580
   [junit4]   2> 61099 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 61099 T167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 61100 T167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683
   [junit4]   2> 61100 T167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/'
   [junit4]   2> 61112 T167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/solr.xml
   [junit4]   2> 61135 T167 oasc.CoreContainer.<init> New CoreContainer 19888098
   [junit4]   2> 61136 T167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/]
   [junit4]   2> 61136 T167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 61136 T167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 61137 T167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 61137 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 61137 T167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 61138 T167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 61138 T167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 61138 T167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 61138 T167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 61139 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 61145 T167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 61146 T167 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54012/solr
   [junit4]   2> 61146 T167 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 61148 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 61151 T323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437f28 name:ZooKeeperConnection Watcher:127.0.0.1:54012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 61151 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 61154 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 61158 T167 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 61159 T325 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e89f0a name:ZooKeeperConnection Watcher:127.0.0.1:54012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 61160 T167 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 61164 T167 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C193 name=collection1 org.apache.solr.core.SolrCore@bf0d24 url=http://127.0.0.1:48632/collection1 node=127.0.0.1:48632_ C193_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:48632_, base_url=http://127.0.0.1:48632}
   [junit4]   2> 61519 T293 C193 P48632 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53821/collection1/ core=collection1 - recoveringAfterStartup=true
   [junit4]   2> 61520 T293 C193 P48632 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48632 START replicas=[http://127.0.0.1:53821/collection1/] nUpdates=100
   [junit4]   2> 61520 T293 C193 P48632 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
   [junit4]   2> 61521 T293 C193 P48632 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
   [junit4]   2> 61521 T293 C193 P48632 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
   [junit4]   2> 61521 T293 C193 P48632 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
   [junit4]   2> 61521 T293 C193 P48632 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 61522 T293 C193 P48632 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53821/collection1/. core=collection1
   [junit4]   2> 61522 T293 C193 P48632 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> ASYNC  NEW_CORE C194 name=collection1 org.apache.solr.core.SolrCore@1f28e1d url=http://127.0.0.1:53821/collection1 node=127.0.0.1:53821_ C194_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53821_, base_url=http://127.0.0.1:53821, leader=true}
   [junit4]   2> 61524 T221 C194 P53821 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
   [junit4]   2> 61527 T218 C194 P53821 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 61528 T218 C194 P53821 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@900e3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d535a0),segFN=segments_1,generation=1}
   [junit4]   2> 61529 T218 C194 P53821 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 61529 T218 C194 P53821 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 61529 T218 C194 P53821 oass.SolrIndexSearcher.<init> Opening Searcher@afdebb realtime
   [junit4]   2> 61530 T218 C194 P53821 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 61530 T218 C194 P53821 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
   [junit4]   2> 61531 T293 C193 P48632 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 61532 T293 C193 P48632 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 61534 T220 C194 P53821 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 61534 T220 C194 P53821 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
   [junit4]   2> 61535 T293 C193 P48632 oasc.RecoveryStrategy.replay No replay needed. core=collection1
   [junit4]   2> 61535 T293 C193 P48632 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
   [junit4]   2> 61535 T293 C193 P48632 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 61535 T293 C193 P48632 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 61537 T293 C193 P48632 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
   [junit4]   2> 61556 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 61557 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node7",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56419_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56419"}
   [junit4]   2> 61561 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node6",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48632_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48632"}
   [junit4]   2> 61566 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61567 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61566 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 61567 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 62010 T234 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 62010 T234 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node7&state=recovering&nodeName=127.0.0.1:56419_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
   [junit4]   2> 62172 T167 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40580_
   [junit4]   2> 62173 T167 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40580_
   [junit4]   2> 62176 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 62176 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 62176 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 62177 T325 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62177 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 62176 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 62178 T189 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62177 T209 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62178 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 62177 T196 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62177 T290 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62178 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 62178 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 62178 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 62180 T225 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62180 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 62181 T273 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62182 T257 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62182 T241 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62182 T307 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 62185 T327 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 62185 T327 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 62186 T327 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 63078 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 63079 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node8",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40580_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40580"}
   [junit4]   2> 63079 T190 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 63079 T190 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 63084 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63085 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63090 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 63186 T327 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 63187 T327 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/collection1
   [junit4]   2> 63187 T327 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 63188 T327 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 63188 T327 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 63190 T327 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/collection1/'
   [junit4]   2> 63190 T327 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/collection1/lib/README' to classloader
   [junit4]   2> 63191 T327 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/collection1/lib/classes/' to classloader
   [junit4]   2> 63211 T327 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 63228 T327 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 63230 T327 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 63235 T327 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 63403 T327 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 63411 T327 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 63412 T327 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 63415 T327 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 63419 T327 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 63431 T327 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 63433 T327 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 63433 T327 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 63433 T327 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 63435 T327 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 63435 T327 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 63440 T327 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 63440 T327 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.schema.TestCloudManagedSchemaAddField-jetty8-1373374107683/collection1/, dataDir=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/
   [junit4]   2> 63441 T327 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1576282
   [junit4]   2> 63442 T327 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8
   [junit4]   2> 63442 T327 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/index/
   [junit4]   2> 63442 T327 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/index' doesn't exist. Creating new index...
   [junit4]   2> 63443 T327 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/index
   [junit4]   2> 63445 T327 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10508fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@bc7387),segFN=segments_1,generation=1}
   [junit4]   2> 63445 T327 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 63448 T327 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 63449 T327 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 63449 T327 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 63450 T327 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 63450 T327 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 63451 T327 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 63451 T327 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 63454 T327 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 63455 T327 oass.SolrIndexSearcher.<init> Opening Searcher@3b507 main
   [junit4]   2> 63455 T327 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 63455 T327 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 63456 T327 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 63461 T328 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b507 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 63463 T327 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 63463 T327 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40580 collection:collection1 shard:shard3
   [junit4]   2> 63463 T327 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 63470 T327 oasc.ZkController.register We are http://127.0.0.1:40580/collection1/ and leader is http://127.0.0.1:54899/collection1/
   [junit4]   2> 63470 T327 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40580
   [junit4]   2> 63470 T327 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 63470 T327 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C195 name=collection1 org.apache.solr.core.SolrCore@1e2fff6 url=http://127.0.0.1:40580/collection1 node=127.0.0.1:40580_ C195_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40580_, base_url=http://127.0.0.1:40580}
   [junit4]   2> 63471 T329 C195 P40580 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 63471 T329 C195 P40580 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 63471 T327 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 63471 T329 C195 P40580 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 63472 T329 C195 P40580 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 63473 T329 C195 P40580 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 63473 T167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 63473 T167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 63474 T167 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 63482 T167 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 63482 T250 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node8, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> ASYNC  NEW_CORE C196 name=collection1 org.apache.solr.core.SolrCore@4ac2c1 url=http://127.0.0.1:47013/collection1 node=127.0.0.1:47013_ C196_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:47013_, base_url=http://127.0.0.1:47013, leader=true}
   [junit4]   2> 63771 T234 C196 P47013 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@f2c82 to URI: /schema
   [junit4]   2> 63804 T234 C196 P47013 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 63828 T234 C196 P47013 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 63866 T307 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63866 T290 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63867 T325 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63867 T241 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63867 T307 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63867 T234 C196 P47013 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Persisted managed schema at /configs/conf1/managed-schema
   [junit4]   2> 63866 T257 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63866 T273 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63866 T189 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63866 T209 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63869 T234 C196 P47013 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=379 
   [junit4]   2> 63866 T225 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 63869 T273 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63869 T257 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63869 T241 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63868 T325 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63867 T290 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63870 T189 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63870 T209 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 63881 T225 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@4ac2c1 url=http://127.0.0.1:47013/collection1 node=127.0.0.1:47013_ C0_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:47013_, base_url=http://127.0.0.1:47013, leader=true}
   [junit4]   2> 27321 T234 C0 P47013 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	47013	PUT	/schema/fields/newfield1	wt=xml	200	-	32	80	http://127.0.0.1:47013	Java/1.8.0-ea	-
   [junit4]   2> 63897 T307 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 30 ms
   [junit4]   2> 63898 T273 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 28 ms
   [junit4]   2> 63904 T209 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 31 ms
   [junit4]   2> 63906 T241 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 34 ms
   [junit4]   2> 63907 T257 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 37 ms
   [junit4]   2> 63910 T290 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 37 ms
   [junit4]   2> 63911 T325 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 40 ms
   [junit4]   2> 63913 T189 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 40 ms
   [junit4]   2> 63917 T225 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 36 ms
   [junit4]   2> ASYNC  NEW_CORE C197 name=collection1 org.apache.solr.core.SolrCore@6fc219 url=http://127.0.0.1:60697/collection1 node=127.0.0.1:60697_ C197_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60697_, base_url=http://127.0.0.1:60697, leader=true}
   [junit4]   2> 63935 T203 C197 P60697 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@21ff09 to URI: /schema
   [junit4]   2> 63939 T203 C197 P60697 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 63940 T203 C197 P60697 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 63942 T203 C197 P60697 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=10 
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@6fc219 url=http://127.0.0.1:60697/collection1 node=127.0.0.1:60697_ C1_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60697_, base_url=http://127.0.0.1:60697, leader=true}
   [junit4]   2> 27383 T203 C1 P60697 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	60697	GET	/schema/fields/newfield1	wt=xml	200	-	0	8	http://127.0.0.1:60697	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C198 name=collection1 org.apache.solr.core.SolrCore@1f28e1d url=http://127.0.0.1:53821/collection1 node=127.0.0.1:53821_ C198_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53821_, base_url=http://127.0.0.1:53821, leader=true}
   [junit4]   2> 63972 T220 C198 P53821 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@826c8 to URI: /schema
   [junit4]   2> 63975 T220 C198 P53821 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 63976 T220 C198 P53821 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 63977 T220 C198 P53821 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=7 
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@1f28e1d url=http://127.0.0.1:53821/collection1 node=127.0.0.1:53821_ C2_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53821_, base_url=http://127.0.0.1:53821, leader=true}
   [junit4]   2> 27416 T220 C2 P53821 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	53821	GET	/schema/fields/newfield1	wt=xml	200	-	0	5	http://127.0.0.1:53821	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C199 name=collection1 org.apache.solr.core.SolrCore@4ac2c1 url=http://127.0.0.1:47013/collection1 node=127.0.0.1:47013_ C199_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:47013_, base_url=http://127.0.0.1:47013, leader=true}
   [junit4]   2> 63988 T237 C199 P47013 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=4 
   [junit4]   2> 27428 T237 C0 P47013 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	47013	GET	/schema/fields/newfield1	wt=xml	200	-	0	4	http://127.0.0.1:47013	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C200 name=collection1 org.apache.solr.core.SolrCore@116295d url=http://127.0.0.1:54899/collection1 node=127.0.0.1:54899_ C200_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:54899_, base_url=http://127.0.0.1:54899, leader=true}
   [junit4]   2> 63998 T253 C200 P54899 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@635f99 to URI: /schema
   [junit4]   2> 63999 T253 C200 P54899 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 64000 T253 C200 P54899 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 64001 T253 C200 P54899 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=5 
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@116295d url=http://127.0.0.1:54899/collection1 node=127.0.0.1:54899_ C3_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:54899_, base_url=http://127.0.0.1:54899, leader=true}
   [junit4]   2> 27440 T253 C3 P54899 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	54899	GET	/schema/fields/newfield1	wt=xml	200	-	0	3	http://127.0.0.1:54899	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C201 name=collection1 org.apache.solr.core.SolrCore@6a59e7 url=http://127.0.0.1:38332/collection1 node=127.0.0.1:38332_ C201_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:38332_, base_url=http://127.0.0.1:38332}
   [junit4]   2> 64008 T268 C201 P38332 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@18c6e0b to URI: /schema
   [junit4]   2> 64010 T268 C201 P38332 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> ASYNC  NEW_CORE C202 name=collection1 org.apache.solr.core.SolrCore@177b244 url=http://127.0.0.1:56419/collection1 node=127.0.0.1:56419_ C202_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:56419_, base_url=http://127.0.0.1:56419}
   [junit4]   2> 64011 T311 C202 P56419 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:47013/collection1/ core=collection1 - recoveringAfterStartup=true
   [junit4]   2> 64011 T268 C201 P38332 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 64011 T311 C202 P56419 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56419 START replicas=[http://127.0.0.1:47013/collection1/] nUpdates=100
   [junit4]   2> 64012 T311 C202 P56419 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
   [junit4]   2> 64012 T311 C202 P56419 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
   [junit4]   2> 64012 T311 C202 P56419 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
   [junit4]   2> 64013 T311 C202 P56419 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
   [junit4]   2> 64012 T268 C201 P38332 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=5 
   [junit4]   2> 64013 T311 C202 P56419 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 64013 T311 C202 P56419 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:47013/collection1/. core=collection1
   [junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@6a59e7 url=http://127.0.0.1:38332/collection1 node=127.0.0.1:38332_ C4_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:38332_, base_url=http://127.0.0.1:38332}
   [junit4]   2> 27452 T268 C4 P38332 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	38332	GET	/schema/fields/newfield1	wt=xml	200	-	0	4	http://127.0.0.1:38332	Java/1.8.0-ea	-
   [junit4]   2> 64014 T311 C202 P56419 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 64014 T236 C199 P47013 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
   [junit4]   2> 64022 T237 C199 P47013 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> ASYNC  NEW_CORE C203 name=collection1 org.apache.solr.core.SolrCore@bf0d24 url=http://127.0.0.1:48632/collection1 node=127.0.0.1:48632_ C203_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:48632_, base_url=http://127.0.0.1:48632}
   [junit4]   2> 64023 T283 C203 P48632 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@186cd8 to URI: /schema
   [junit4]   2> 64025 T237 C199 P47013 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@124bc24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@154d388),segFN=segments_1,generation=1}
   [junit4]   2> 64026 T237 C199 P47013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 64026 T283 C203 P48632 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 64026 T237 C199 P47013 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 64027 T283 C203 P48632 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 64027 T237 C199 P47013 oass.SolrIndexSearcher.<init> Opening Searcher@82c58e realtime
   [junit4]   2> 64027 T237 C199 P47013 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 64028 T283 C203 P48632 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=7 
   [junit4]   2> 64028 T237 C199 P47013 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
   [junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@bf0d24 url=http://127.0.0.1:48632/collection1 node=127.0.0.1:48632_ C5_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:48632_, base_url=http://127.0.0.1:48632}
   [junit4]   2> 27466 T283 C5 P48632 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	48632	GET	/schema/fields/newfield1	wt=xml	200	-	0	3	http://127.0.0.1:48632	Java/1.8.0-ea	-
   [junit4]   2> 64029 T311 C202 P56419 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 64030 T311 C202 P56419 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 64038 T236 C199 P47013 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 64038 T236 C199 P47013 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3 
   [junit4]   2> 64039 T303 C202 P56419 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@135bba7 to URI: /schema
   [junit4]   2> 64039 T311 C202 P56419 oasc.RecoveryStrategy.replay No replay needed. core=collection1
   [junit4]   2> 64040 T311 C202 P56419 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
   [junit4]   2> 64040 T311 C202 P56419 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 64040 T311 C202 P56419 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 64041 T303 C202 P56419 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 64041 T311 C202 P56419 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
   [junit4]   2> 64042 T303 C202 P56419 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 64043 T303 C202 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=8 
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@177b244 url=http://127.0.0.1:56419/collection1 node=127.0.0.1:56419_ C6_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:56419_, base_url=http://127.0.0.1:56419}
   [junit4]   2> 27481 T303 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield1	wt=xml	200	-	0	3	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C204 name=collection1 org.apache.solr.core.SolrCore@1e2fff6 url=http://127.0.0.1:40580/collection1 node=127.0.0.1:40580_ C204_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40580_, base_url=http://127.0.0.1:40580}
   [junit4]   2> 64050 T318 C204 P40580 oejsh.ContextHandler$Context.log SolrRestApi: [Restlet] Attaching application: org.apache.solr.rest.SolrRestApi@d277f8 to URI: /schema
   [junit4]   2> ASYNC  NEW_CORE C205 name=collection1 org.apache.solr.core.SolrCore@1e2fff6 url=http://127.0.0.1:40580/collection1 node=127.0.0.1:40580_ C205_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40580_, base_url=http://127.0.0.1:40580}
   [junit4]   2> 64061 T318 C205 P40580 oasr.SolrRestApi.createInboundRoot createInboundRoot started
   [junit4]   2> 64061 T318 C205 P40580 oasr.SolrRestApi.createInboundRoot createInboundRoot complete
   [junit4]   2> 64062 T318 C205 P40580 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield1 params={wt=xml} status=0 QTime=14 
   [junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@1e2fff6 url=http://127.0.0.1:40580/collection1 node=127.0.0.1:40580_ C7_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40580_, base_url=http://127.0.0.1:40580}
   [junit4]   2> 27501 T318 C7 P40580 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	40580	GET	/schema/fields/newfield1	wt=xml	200	-	0	12	http://127.0.0.1:40580	Java/1.8.0-ea	-
   [junit4]   2> 64073 T225 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T257 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T325 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T290 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T209 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64075 T257 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64075 T290 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> ASYNC  NEW_CORE C206 name=collection1 org.apache.solr.core.SolrCore@1f28e1d url=http://127.0.0.1:53821/collection1 node=127.0.0.1:53821_ C206_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53821_, base_url=http://127.0.0.1:53821, leader=true}
   [junit4]   2> 64074 T221 C206 P53821 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Persisted managed schema at /configs/conf1/managed-schema
   [junit4]   2> 64074 T307 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T241 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T273 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64074 T189 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64077 T225 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64077 T307 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64076 T221 C206 P53821 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=9 
   [junit4]   2> 27517 T221 C2 P53821 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	53821	PUT	/schema/fields/newfield2	wt=xml	200	-	32	11	http://127.0.0.1:53821	Java/1.8.0-ea	-
   [junit4]   2> 64075 T325 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64081 T189 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64086 T257 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 11 ms
   [junit4]   2> 64086 T209 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64078 T273 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64078 T241 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64092 T189 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 11 ms
   [junit4]   2> 64086 T290 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 11 ms
   [junit4]   2> ASYNC  NEW_CORE C207 name=collection1 org.apache.solr.core.SolrCore@6fc219 url=http://127.0.0.1:60697/collection1 node=127.0.0.1:60697_ C207_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60697_, base_url=http://127.0.0.1:60697, leader=true}
   [junit4]   2> 64098 T202 C207 P60697 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} msg={Field 'newfield2' not found.} status=404 QTime=4 
   [junit4]   2> 27577 T202 C1 P60697 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	60697	GET	/schema/fields/newfield2	wt=xml	404	-	0	42	http://127.0.0.1:60697	Java/1.8.0-ea	-
   [junit4]   2> 64140 T325 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 60 ms
   [junit4]   2> 64145 T307 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 68 ms
   [junit4]   2> 64092 T225 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 15 ms
   [junit4]   2> 64147 T209 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 60 ms
   [junit4]   2> 64148 T273 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 60 ms
   [junit4]   2> 64154 T241 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 66 ms
   [junit4]   2> 64155 T205 C207 P60697 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=2 
   [junit4]   2> 27593 T205 C1 P60697 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	60697	GET	/schema/fields/newfield2	wt=xml	200	-	0	1	http://127.0.0.1:60697	Java/1.8.0-ea	-
   [junit4]   2> 64159 T167 oass.TestCloudManagedSchemaAddField.doTest On attempt #2, successful request /schema/fields/newfield2?wt=xml against server http://127.0.0.1:60697/collection1 after 67 ms
   [junit4]   2> ASYNC  NEW_CORE C208 name=collection1 org.apache.solr.core.SolrCore@1f28e1d url=http://127.0.0.1:53821/collection1 node=127.0.0.1:53821_ C208_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53821_, base_url=http://127.0.0.1:53821, leader=true}
   [junit4]   2> 64161 T218 C208 P53821 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=0 
   [junit4]   2> 27600 T218 C2 P53821 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	53821	GET	/schema/fields/newfield2	wt=xml	200	-	0	1	http://127.0.0.1:53821	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C209 name=collection1 org.apache.solr.core.SolrCore@4ac2c1 url=http://127.0.0.1:47013/collection1 node=127.0.0.1:47013_ C209_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:47013_, base_url=http://127.0.0.1:47013, leader=true}
   [junit4]   2> 64166 T234 C209 P47013 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=0 
   [junit4]   2> 27605 T234 C0 P47013 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	47013	GET	/schema/fields/newfield2	wt=xml	200	-	0	1	http://127.0.0.1:47013	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C210 name=collection1 org.apache.solr.core.SolrCore@116295d url=http://127.0.0.1:54899/collection1 node=127.0.0.1:54899_ C210_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:54899_, base_url=http://127.0.0.1:54899, leader=true}
   [junit4]   2> 64172 T252 C210 P54899 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27611 T252 C3 P54899 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	54899	GET	/schema/fields/newfield2	wt=xml	200	-	0	1	http://127.0.0.1:54899	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C211 name=collection1 org.apache.solr.core.SolrCore@6a59e7 url=http://127.0.0.1:38332/collection1 node=127.0.0.1:38332_ C211_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:38332_, base_url=http://127.0.0.1:38332}
   [junit4]   2> 64178 T267 C211 P38332 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27617 T267 C4 P38332 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	38332	GET	/schema/fields/newfield2	wt=xml	200	-	0	2	http://127.0.0.1:38332	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C212 name=collection1 org.apache.solr.core.SolrCore@bf0d24 url=http://127.0.0.1:48632/collection1 node=127.0.0.1:48632_ C212_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:48632_, base_url=http://127.0.0.1:48632}
   [junit4]   2> 64184 T286 C212 P48632 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27623 T286 C5 P48632 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	48632	GET	/schema/fields/newfield2	wt=xml	200	-	0	2	http://127.0.0.1:48632	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C213 name=collection1 org.apache.solr.core.SolrCore@177b244 url=http://127.0.0.1:56419/collection1 node=127.0.0.1:56419_ C213_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:56419_, base_url=http://127.0.0.1:56419}
   [junit4]   2> 64192 T300 C213 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=2 
   [junit4]   2> 27631 T300 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield2	wt=xml	200	-	0	2	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C214 name=collection1 org.apache.solr.core.SolrCore@1e2fff6 url=http://127.0.0.1:40580/collection1 node=127.0.0.1:40580_ C214_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40580_, base_url=http://127.0.0.1:40580}
   [junit4]   2> 64200 T319 C214 P40580 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield2 params={wt=xml} status=0 QTime=2 
   [junit4]   2> 27639 T319 C7 P40580 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	40580	GET	/schema/fields/newfield2	wt=xml	200	-	0	1	http://127.0.0.1:40580	Java/1.8.0-ea	-
   [junit4]   2> 64214 T209 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64214 T290 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64214 T225 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64214 T325 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64215 T257 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64214 T189 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64215 T235 C209 P47013 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Persisted managed schema at /configs/conf1/managed-schema
   [junit4]   2> 64215 T290 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64215 T209 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64214 T241 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64214 T307 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64214 T273 oass.ZkIndexSchemaReader$1.process A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 64216 T189 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64219 T235 C209 P47013 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=12 
   [junit4]   2> 27658 T235 C0 P47013 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	47013	PUT	/schema/fields/newfield3	wt=xml	200	-	32	12	http://127.0.0.1:47013	Java/1.8.0-ea	-
   [junit4]   2> 64222 T273 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64223 T241 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64225 T203 C207 P60697 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27681 T203 C1 P60697 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	60697	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:60697	Java/1.8.0-ea	-
   [junit4]   2> 64216 T325 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64216 T257 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64215 T225 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64254 T241 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 31 ms
   [junit4]   2> 64245 T189 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 26 ms
   [junit4]   2> 64256 T273 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 33 ms
   [junit4]   2> ASYNC  NEW_CORE C215 name=collection1 org.apache.solr.core.SolrCore@6fc219 url=http://127.0.0.1:60697/collection1 node=127.0.0.1:60697_ C215_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60697_, base_url=http://127.0.0.1:60697, leader=true}
   [junit4]   2> 64258 T204 C215 P60697 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27697 T204 C1 P60697 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	60697	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:60697	Java/1.8.0-ea	-
   [junit4]   2> 64263 T225 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 9 ms
   [junit4]   2> 64264 T257 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 10 ms
   [junit4]   2> 64265 T209 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 47 ms
   [junit4]   2> 64267 T290 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 50 ms
   [junit4]   2> 64265 T325 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 18 ms
   [junit4]   2> 64275 T202 C215 P60697 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27713 T202 C1 P60697 orel.LogFilter.afterHandle 2013-07-09	12:48:30	127.0.0.1	-	127.0.0.1	60697	GET	/schema/fields/newfield3	wt=xml	200	-	0	1	http://127.0.0.1:60697	Java/1.8.0-ea	-
   [junit4]   2> 64278 T167 oass.TestCloudManagedSchemaAddField.doTest On attempt #3, successful request /schema/fields/newfield3?wt=xml against server http://127.0.0.1:60697/collection1 after 56 ms
   [junit4]   2> ASYNC  NEW_CORE C216 name=collection1 org.apache.solr.core.SolrCore@1f28e1d url=http://127.0.0.1:53821/collection1 node=127.0.0.1:53821_ C216_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53821_, base_url=http://127.0.0.1:53821, leader=true}
   [junit4]   2> 64328 T219 C216 P53821 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=48 
   [junit4]   2> 27767 T219 C2 P53821 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	53821	GET	/schema/fields/newfield3	wt=xml	200	-	0	49	http://127.0.0.1:53821	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C217 name=collection1 org.apache.solr.core.SolrCore@4ac2c1 url=http://127.0.0.1:47013/collection1 node=127.0.0.1:47013_ C217_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:47013_, base_url=http://127.0.0.1:47013, leader=true}
   [junit4]   2> 64335 T237 C217 P47013 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27774 T237 C0 P47013 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	47013	GET	/schema/fields/newfield3	wt=xml	200	-	0	2	http://127.0.0.1:47013	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C218 name=collection1 org.apache.solr.core.SolrCore@116295d url=http://127.0.0.1:54899/collection1 node=127.0.0.1:54899_ C218_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:54899_, base_url=http://127.0.0.1:54899, leader=true}
   [junit4]   2> 64344 T251 C218 P54899 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27783 T251 C3 P54899 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	54899	GET	/schema/fields/newfield3	wt=xml	200	-	0	2	http://127.0.0.1:54899	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C219 name=collection1 org.apache.solr.core.SolrCore@6a59e7 url=http://127.0.0.1:38332/collection1 node=127.0.0.1:38332_ C219_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:38332_, base_url=http://127.0.0.1:38332}
   [junit4]   2> 64350 T269 C219 P38332 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=1 
   [junit4]   2> 27789 T269 C4 P38332 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	38332	GET	/schema/fields/newfield3	wt=xml	200	-	0	2	http://127.0.0.1:38332	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C220 name=collection1 org.apache.solr.core.SolrCore@bf0d24 url=http://127.0.0.1:48632/collection1 node=127.0.0.1:48632_ C220_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:48632_, base_url=http://127.0.0.1:48632}
   [junit4]   2> 64355 T284 C220 P48632 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} status=0 QTime=0 
   [junit4]   2> 27794 T284 C5 P48632 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	48632	GET	/schema/fields/newfield3	wt=xml	200	-	0	1	http://127.0.0.1:48632	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C221 name=collection1 org.apache.solr.core.SolrCore@177b244 url=http://127.0.0.1:56419/collection1 node=127.0.0.1:56419_ C221_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:56419_, base_url=http://127.0.0.1:56419}
   [junit4]   2> 64360 T301 C221 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27799 T301 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64375 T302 C221 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27814 T302 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64391 T303 C221 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27830 T303 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	2	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64407 T300 C221 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27846 T300 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> ASYNC  NEW_CORE C222 name=collection1 org.apache.solr.core.SolrCore@177b244 url=http://127.0.0.1:56419/collection1 node=127.0.0.1:56419_ C222_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:56419_, base_url=http://127.0.0.1:56419}
   [junit4]   2> 64425 T302 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=0 
   [junit4]   2> 27864 T302 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64440 T303 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27879 T303 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64454 T301 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=0 
   [junit4]   2> 27893 T301 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64471 T302 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=3 
   [junit4]   2> 27909 T302 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	3	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64486 T300 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27925 T300 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64504 T303 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=0 
   [junit4]   2> 27943 T303 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64520 T301 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27959 T301 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64535 T302 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 27974 T302 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64550 T300 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=0 
   [junit4]   2> 27989 T300 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64564 T303 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 28003 T303 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64578 T301 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=0 
   [junit4]   2> 28017 T301 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64593 T302 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 28031 T302 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64595 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 64596 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node8",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40580_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40580"}
   [junit4]   2> 64600 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node7",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56419_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56419"}
   [junit4]   2> 64606 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64606 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64607 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64607 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64607 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64607 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64607 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64607 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64608 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 64608 T300 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 28047 T300 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	2	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64624 T303 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 28063 T303 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64640 T301 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 28078 T301 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64654 T302 C222 P56419 oasrs.BaseSchemaResource.handlePostExecution [collection1] webapp=null path=/schema/fields/newfield3 params={wt=xml} msg={Field 'newfield3' not found.} status=404 QTime=1 
   [junit4]   2> 28093 T302 C6 P56419 orel.LogFilter.afterHandle 2013-07-09	12:48:31	127.0.0.1	-	127.0.0.1	56419	GET	/schema/fields/newfield3	wt=xml	404	-	0	1	http://127.0.0.1:56419	Java/1.8.0-ea	-
   [junit4]   2> 64656 T167 oass.TestCloudManagedSchemaAddField.doTest ERROR Max retry count 20 exceeded after 434 ms.  QUERY FAILED: xpath=/response/lst[@name='responseHeader']/int[@name='status'][.='0']  request=/schema/fields/newfield3?wt=xml  response=<?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> 	<response>
   [junit4]   2> 	
   [junit4]   2> 	<lst name="responseHeader">
   [junit4]   2> 	  <int name="status">404</int>
   [junit4]   2> 	  <int name="QTime">1</int>
   [junit4]   2> 	</lst>
   [junit4]   2> 	<lst name="error">
   [junit4]   2> 	  <str name="msg">Field 'newfield3' not found.</str>
   [junit4]   2> 	  <int name="code">404</int>
   [junit4]   2> 	</lst>
   [junit4]   2> 	</response>
   [junit4]   2> 	
   [junit4]   2> 64661 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 59178
   [junit4]   2> 64662 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=28258397
   [junit4]   2> 64744 T307 oass.ZkIndexSchemaReader.updateSchema Retrieved schema from ZooKeeper
   [junit4]   2> 64754 T307 oass.ZkIndexSchemaReader.updateSchema Finished refreshing schema in 10 ms
   [junit4]   2> 64755 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 65483 T250 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node8, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
   [junit4]   2> 65483 T250 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node8&state=recovering&nodeName=127.0.0.1:40580_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001 
   [junit4]   2> 66115 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 66116 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:59178_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59178"}
   [junit4]   2> 66116 T190 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 66120 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66121 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66121 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66121 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66121 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66121 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66121 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66120 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 66120 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> ASYNC  NEW_CORE C223 name=collection1 org.apache.solr.core.SolrCore@1e2fff6 url=http://127.0.0.1:40580/collection1 node=127.0.0.1:40580_ C223_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:40580_, base_url=http://127.0.0.1:40580}
   [junit4]   2> 67484 T329 C223 P40580 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54899/collection1/ core=collection1 - recoveringAfterStartup=true
   [junit4]   2> 67485 T329 C223 P40580 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40580 START replicas=[http://127.0.0.1:54899/collection1/] nUpdates=100
   [junit4]   2> 67485 T329 C223 P40580 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
   [junit4]   2> 67485 T329 C223 P40580 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
   [junit4]   2> 67485 T329 C223 P40580 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
   [junit4]   2> 67486 T329 C223 P40580 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
   [junit4]   2> 67487 T329 C223 P40580 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 67487 T329 C223 P40580 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54899/collection1/. core=collection1
   [junit4]   2> ASYNC  NEW_CORE C224 name=collection1 org.apache.solr.core.SolrCore@116295d url=http://127.0.0.1:54899/collection1 node=127.0.0.1:54899_ C224_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:54899_, base_url=http://127.0.0.1:54899, leader=true}
   [junit4]   2> 67487 T251 C224 P54899 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
   [junit4]   2> 67489 T329 C223 P40580 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 67495 T252 C224 P54899 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 67498 T252 C224 P54899 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15b2994 lockFactory=org.apache.lucene.store.NativeFSLockFactory@147d3d7),segFN=segments_1,generation=1}
   [junit4]   2> 67498 T252 C224 P54899 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 67498 T252 C224 P54899 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 67499 T252 C224 P54899 oass.SolrIndexSearcher.<init> Opening Searcher@119bdfe realtime
   [junit4]   2> 67499 T252 C224 P54899 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 67499 T252 C224 P54899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
   [junit4]   2> 67501 T329 C223 P40580 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 67501 T329 C223 P40580 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 67503 T250 C224 P54899 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 67504 T250 C224 P54899 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
   [junit4]   2> 67505 T329 C223 P40580 oasc.RecoveryStrategy.replay No replay needed. core=collection1
   [junit4]   2> 67505 T329 C223 P40580 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
   [junit4]   2> 67505 T329 C223 P40580 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 67505 T329 C223 P40580 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 67506 T329 C223 P40580 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
   [junit4]   2> 67626 T190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 67627 T190 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node8",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40580_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40580"}
   [junit4]   2> 67632 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67632 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67633 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67633 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67632 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67633 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67633 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67633 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67633 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67663 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 67664 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 67670 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@621e2d
   [junit4]   2> 67677 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 67678 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 67678 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 67679 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 67679 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 67680 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 67680 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/index;done=false>>]
   [junit4]   2> 67681 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data/index
   [junit4]   2> 67681 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data;done=false>>]
   [junit4]   2> 67681 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/control/data
   [junit4]   2> 67683 T190 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491139-127.0.0.1:59178_-n_0000000000) am no longer a leader.
   [junit4]   2> 67684 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67685 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67688 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67688 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67689 T290 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67689 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67689 T225 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67689 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67690 T189 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 67690 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 67689 T325 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67691 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67691 T241 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67691 T257 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67692 T307 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67691 T189 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 67690 T209 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67693 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67694 T273 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 67695 T209 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 67697 T209 oasc.Overseer.start Overseer (id=90005443926491142-127.0.0.1:60697_-n_0000000001) starting
   [junit4]   2> 67702 T335 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 67703 T334 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 67705 T334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 67705 T334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59178_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59178"}
   [junit4]   2> 67711 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 67713 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67714 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67714 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67714 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67714 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67713 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67713 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67713 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 67763 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60697
   [junit4]   2> 67763 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=19210768
   [junit4]   2> 68765 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 68766 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 68767 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6fc219
   [junit4]   2> 68771 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 68771 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 68771 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 68772 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 68773 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 68774 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 68774 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/index;done=false>>]
   [junit4]   2> 68774 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1/index
   [junit4]   2> 68774 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1;done=false>>]
   [junit4]   2> 68775 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty1
   [junit4]   2> 68776 T334 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491142-127.0.0.1:60697_-n_0000000001) am no longer a leader.
   [junit4]   2> 68777 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 68777 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 68777 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 68778 T290 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68778 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68778 T325 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68779 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68779 T225 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 68779 T273 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 68780 T307 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68782 T257 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68783 T241 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68786 T225 oasc.Overseer.start Overseer (id=90005443926491144-127.0.0.1:53821_-n_0000000002) starting
   [junit4]   2> 68787 T273 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
   [junit4]   2> 68787 T273 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
   [junit4]   2> 68787 T273 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 68787 T273 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38332/collection1/
   [junit4]   2> 68788 T273 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38332 START replicas=[http://127.0.0.1:60697/collection1/] nUpdates=100
   [junit4]   2> 68793 T273 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38332 DONE.  We have no versions.  sync failed.
   [junit4]   2> 68794 T273 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 68794 T273 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 68794 T273 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38332/collection1/ shard1
   [junit4]   2> 68794 T273 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 68797 T338 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 68799 T225 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68799 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68800 T337 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 68800 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 68802 T273 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 68802 T337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 68803 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60697_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60697"}
   [junit4]   2> 68803 T337 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 68808 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 68815 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60697_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60697"}
   [junit4]   2> 68826 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68826 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68826 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68827 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68827 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68826 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68827 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 68860 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53821
   [junit4]   2> 68860 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=20382611
   [junit4]   2> 69862 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 69862 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 69864 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1f28e1d
   [junit4]   2> 69867 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 69868 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 69868 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 69868 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 69870 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 69870 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 69870 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2;done=false>>]
   [junit4]   2> 69871 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2
   [junit4]   2> 69871 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/index;done=false>>]
   [junit4]   2> 69871 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty2/index
   [junit4]   2> 69872 T337 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491144-127.0.0.1:53821_-n_0000000002) am no longer a leader.
   [junit4]   2> 69873 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 69874 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 69874 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 69875 T290 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 69875 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69875 T325 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 69875 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69880 T273 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 69880 T307 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 69881 T241 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 69881 T257 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 69882 T290 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard4
   [junit4]   2> 69885 T241 oasc.Overseer.start Overseer (id=90005443926491146-127.0.0.1:47013_-n_0000000003) starting
   [junit4]   2> 69887 T290 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
   [junit4]   2> 69887 T290 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
   [junit4]   2> 69887 T290 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 69888 T290 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48632/collection1/
   [junit4]   2> 69888 T290 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 69888 T290 oasc.SyncStrategy.syncToMe http://127.0.0.1:48632/collection1/ has no replicas
   [junit4]   2> 69888 T290 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48632/collection1/ shard4
   [junit4]   2> 69889 T290 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard4
   [junit4]   2> 69891 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 69891 T340 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 69894 T241 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 69895 T339 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 69897 T339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 69898 T339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:53821_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53821"}
   [junit4]   2> 69898 T339 oasc.Overseer$ClusterStateUpdater.updateState shard=shard4 is already registered
   [junit4]   2> 69902 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 69902 T339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53821_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53821"}
   [junit4]   2> 69912 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69912 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69912 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69912 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69912 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69912 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 69953 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47013
   [junit4]   2> 69953 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=18286318
   [junit4]   2> 70077 T209 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 70078 T209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 70078 T209 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 70955 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 70955 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 70957 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4ac2c1
   [junit4]   2> 70960 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 70961 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 70961 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 70961 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 70963 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 70963 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 70963 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3;done=false>>]
   [junit4]   2> 70964 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3
   [junit4]   2> 70964 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/index;done=false>>]
   [junit4]   2> 70964 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty3/index
   [junit4]   2> 70965 T339 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491146-127.0.0.1:47013_-n_0000000003) am no longer a leader.
   [junit4]   2> 70967 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 70967 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 70967 T241 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 70967 T241 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 70968 T290 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 70968 T325 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 70968 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 70968 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 70969 T307 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 70969 T257 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 70970 T273 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 70971 T307 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
   [junit4]   2> 70971 T307 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
   [junit4]   2> 70971 T307 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 70971 T307 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56419/collection1/
   [junit4]   2> 70972 T307 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56419 START replicas=[http://127.0.0.1:47013/collection1/] nUpdates=100
   [junit4]   2> 70972 T307 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56419 DONE.  We have no versions.  sync failed.
   [junit4]   2> 70972 T257 oasc.Overseer.start Overseer (id=90005443926491148-127.0.0.1:54899_-n_0000000004) starting
   [junit4]   2> 70972 T307 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 70972 T307 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 70973 T307 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56419/collection1/ shard2
   [junit4]   2> 70976 T307 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 70980 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 70980 T343 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 70983 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
   [junit4]   2> 70983 T257 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 70984 T342 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 70985 T307 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 70986 T342 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 70987 T342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:47013_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47013"}
   [junit4]   2> 70987 T342 oasc.Overseer$ClusterStateUpdater.updateState shard=shard2 is already registered
   [junit4]   2> 70990 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 70991 T342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47013_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47013"}
   [junit4]   2> 71001 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 71002 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 71002 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 71002 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 71002 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 71041 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 54899
   [junit4]   2> 71042 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=4421876
   [junit4]   2> 71176 T225 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 71176 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 71176 T225 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 72043 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 72044 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 72045 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@116295d
   [junit4]   2> 72049 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 72050 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 72050 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 72050 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 72053 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 72053 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 72054 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/index;done=false>>]
   [junit4]   2> 72054 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4/index
   [junit4]   2> 72054 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4;done=false>>]
   [junit4]   2> 72055 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty4
   [junit4]   2> 72056 T342 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491148-127.0.0.1:54899_-n_0000000004) am no longer a leader.
   [junit4]   2> 72057 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 72057 T257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 72058 T290 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 72059 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 72059 T325 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 72060 T273 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 72061 T307 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 72062 T325 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
   [junit4]   2> 72062 T325 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
   [junit4]   2> 72062 T325 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 72062 T325 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40580/collection1/
   [junit4]   2> 72062 T273 oasc.Overseer.start Overseer (id=90005443926491150-127.0.0.1:38332_-n_0000000005) starting
   [junit4]   2> 72062 T325 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40580 START replicas=[http://127.0.0.1:54899/collection1/] nUpdates=100
   [junit4]   2> 72063 T325 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40580 DONE.  We have no versions.  sync failed.
   [junit4]   2> 72063 T325 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 72063 T325 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 72063 T325 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40580/collection1/ shard3
   [junit4]   2> 72064 T325 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 72067 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 72067 T346 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 72070 T273 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 72071 T325 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 72071 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 72071 T345 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 72073 T345 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 72074 T345 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node4",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:54899_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54899"}
   [junit4]   2> 72074 T345 oasc.Overseer$ClusterStateUpdater.updateState shard=shard3 is already registered
   [junit4]   2> 72077 T345 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node4",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54899_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54899"}
   [junit4]   2> 72081 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 72087 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 72087 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 72087 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 72087 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 72132 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 38332
   [junit4]   2> 72133 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=20888152
   [junit4]   2> 73134 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 73135 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 73136 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6a59e7
   [junit4]   2> 73141 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 73141 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 73142 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 73142 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 73143 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 73143 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 73144 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5;done=false>>]
   [junit4]   2> 73144 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5
   [junit4]   2> 73144 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/index;done=false>>]
   [junit4]   2> 73145 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty5/index
   [junit4]   2> 73146 T345 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491150-127.0.0.1:38332_-n_0000000005) am no longer a leader.
   [junit4]   2> 73147 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 73148 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 73148 T273 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 73148 T290 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73148 T273 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 73148 T325 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73149 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73149 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73153 T307 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73158 T290 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 73162 T290 oasc.Overseer.start Overseer (id=90005443926491152-127.0.0.1:48632_-n_0000000006) starting
   [junit4]   2> 73168 T348 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 73168 T347 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 73173 T347 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 73174 T347 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node5",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:38332_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38332"}
   [junit4]   2> 73174 T347 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 73182 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 73184 T347 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node5",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38332_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38332"}
   [junit4]   2> 73191 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73192 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73191 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73234 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 48632
   [junit4]   2> 73234 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=14118405
   [junit4]   2> 73358 T257 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 73358 T257 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 74236 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 74236 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 74239 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@bf0d24
   [junit4]   2> 74243 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 74244 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 74244 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 74244 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 74245 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 74245 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 74245 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6;done=false>>]
   [junit4]   2> 74246 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6
   [junit4]   2> 74246 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/index;done=false>>]
   [junit4]   2> 74246 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty6/index
   [junit4]   2> 74247 T347 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491152-127.0.0.1:48632_-n_0000000006) am no longer a leader.
   [junit4]   2> 74248 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74249 T325 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 74250 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 74251 T307 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 74254 T307 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 74256 T307 oasc.Overseer.start Overseer (id=90005443926491154-127.0.0.1:56419_-n_0000000007) starting
   [junit4]   2> 74260 T350 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 74261 T349 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 74263 T349 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 74263 T349 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node6",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:48632_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48632"}
   [junit4]   2> 74264 T349 oasc.Overseer$ClusterStateUpdater.updateState shard=shard4 is already registered
   [junit4]   2> 74267 T349 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node6",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48632_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48632"}
   [junit4]   2> 74271 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 74272 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 74274 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 74323 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 56419
   [junit4]   2> 74323 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=13873096
   [junit4]   2> 75325 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 75325 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 75327 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@177b244
   [junit4]   2> 75330 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 75330 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 75331 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 75331 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 75332 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 75332 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 75333 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7;done=false>>]
   [junit4]   2> 75333 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7
   [junit4]   2> 75333 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/index;done=false>>]
   [junit4]   2> 75334 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty7/index
   [junit4]   2> 75335 T349 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491154-127.0.0.1:56419_-n_0000000007) am no longer a leader.
   [junit4]   2> 75337 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 75338 T325 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 75340 T325 oasc.Overseer.start Overseer (id=90005443926491156-127.0.0.1:40580_-n_0000000008) starting
   [junit4]   2> 75344 T352 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 75345 T325 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 75345 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75346 T351 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 75349 T351 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75350 T351 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node7",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:56419_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56419"}
   [junit4]   2> 75350 T351 oasc.Overseer$ClusterStateUpdater.updateState shard=shard2 is already registered
   [junit4]   2> 75353 T351 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node7",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56419_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56419"}
   [junit4]   2> 75359 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75366 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 75417 T167 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 40580
   [junit4]   2> 75418 T167 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=19888098
   [junit4]   2> 75548 T290 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 75549 T290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75549 T290 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 76420 T167 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 76420 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 76421 T167 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1e2fff6
   [junit4]   2> 76430 T167 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 76430 T167 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 76431 T167 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 76431 T167 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 76431 T167 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 76432 T167 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 76432 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/index [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/index;done=false>>]
   [junit4]   2> 76432 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8/index
   [junit4]   2> 76433 T167 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8 [CachedDir<<refCount=0;path=./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8;done=false>>]
   [junit4]   2> 76433 T167 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.schema.TestCloudManagedSchemaAddField-1373374083241/jetty8
   [junit4]   2> 76434 T351 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90005443926491156-127.0.0.1:40580_-n_0000000008) am no longer a leader.
   [junit4]   2> 76456 T167 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 76513 T167 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 76515 T167 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54012 54012
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCloudManagedSchemaAddField -Dtests.method=testDistribSearch -Dtests.seed=141EDF8B23989943 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr -Dtests.timezone=Africa/Lome -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 40.0s J0 | TestCloudManagedSchemaAddField.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Max retry count 20 exceeded after 434 ms.  QUERY FAILED: xpath=/response/lst[@name='responseHeader']/int[@name='status'][.='0']  request=/schema/fields/newfield3?wt=xml  response=<?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader">
   [junit4]    >   <int name="status">404</int>
   [junit4]    >   <int name="QTime">1</int>
   [junit4]    > </lst>
   [junit4]    > <lst name="error">
   [junit4]    >   <str name="msg">Field 'newfield3' not found.</str>
   [junit4]    >   <int name="code">404</int>
   [junit4]    > </lst>
   [junit4]    > </response>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([141EDF8B23989943:95F8519354C7F97F]:0)
   [junit4]    > 	at org.apache.solr.schema.TestCloudManagedSchemaAddField.doTest(TestCloudManagedSchemaAddField.java:119)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 76550 T167 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 39991 T166 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> 76637 T307 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 76638 T307 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 77735 T325 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 77735 T325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 77735 T325 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=sr, timezone=Africa/Lome
   [junit4]   2> NOTE: Linux 3.2.0-49-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=150763160,total=193617920
   [junit4]   2> NOTE: All tests run in this JVM: [BlockCacheTest, TestStressRecovery, ReturnFieldsTest, TestRemoteStreaming, RAMDirectoryFactoryTest, DistributedQueryElevationComponentTest, BlockDirectoryTest, SolrCoreTest, TestIBSimilarityFactory, TestPseudoReturnFields, DateFieldTest, TestLFUCache, TestCloudManagedSchemaAddField]
   [junit4] Completed on J0 in 41.27s, 1 test, 1 failure <<< FAILURES!

[...truncated 863 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:369: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:890: There were test failures: 316 suites, 1333 tests, 1 failure, 23 ignored (10 assumptions)

Total time: 35 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b96 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message