lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_25) - Build # 6289 - Failure!
Date Mon, 01 Jul 2013 13:33:57 GMT
I've got to look closer at this - offhand a fail like this looks odd. It says that if you search the control cluster, it has 99 docs, if you search the SolrCloud cluster, it has 99 docs, but when you sum the docs from one random replica in each shard and add them, you get 55 docs (searches with distrib=false). What's weird about that is that a consistency check of the replicas has already passed - so you would not expect any way to add up docs on a replica of each shard and get less than 99 unless there was some bug in the test assertion logic or something…

- Mark

On Jun 29, 2013, at 5:13 AM, Policeman Jenkins Server <jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6289/
> Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseParallelGC
> 
> 1 tests failed.
> REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
> 
> Error Message:
> document count mismatch.  control=99 sum(shards)=55 cloudClient=99
> 
> Stack Trace:
> java.lang.AssertionError: document count mismatch.  control=99 sum(shards)=55 cloudClient=99
> 	at __randomizedtesting.SeedInfo.seed([72DE2FB2FAE961FB:F338A1AA8DB601C7]:0)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1147)
> 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:194)
> 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> 	at java.lang.Thread.run(Thread.java:724)
> 
> 
> 
> 
> Build Log:
> [...truncated 10158 lines...]
> [junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
> [junit4:junit4]   2> 981238 T4065 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /rd/u
> [junit4:junit4]   2> 981243 T4065 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
> [junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyNothingIsSafeTest-1372497033445
> [junit4:junit4]   2> 981243 T4065 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4:junit4]   2> 981244 T4066 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4:junit4]   2> 981344 T4065 oasc.ZkTestServer.run start zk server on port:59208
> [junit4:junit4]   2> 981345 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 981431 T4072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d9294f7 name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 981431 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 981431 T4065 oascc.SolrZkClient.makePath makePath: /solr
> [junit4:junit4]   2> 981435 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 981437 T4074 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b4c5b64 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 981437 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 981437 T4065 oascc.SolrZkClient.makePath makePath: /collections/collection1
> [junit4:junit4]   2> 981439 T4065 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
> [junit4:junit4]   2> 981441 T4065 oascc.SolrZkClient.makePath makePath: /collections/control_collection
> [junit4:junit4]   2> 981442 T4065 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
> [junit4:junit4]   2> 981443 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
> [junit4:junit4]   2> 981444 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
> [junit4:junit4]   2> 981447 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
> [junit4:junit4]   2> 981447 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
> [junit4:junit4]   2> 981449 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
> [junit4:junit4]   2> 981449 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
> [junit4:junit4]   2> 981451 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
> [junit4:junit4]   2> 981451 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
> [junit4:junit4]   2> 981453 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
> [junit4:junit4]   2> 981453 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
> [junit4:junit4]   2> 981455 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
> [junit4:junit4]   2> 981455 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
> [junit4:junit4]   2> 981457 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4]   2> 981457 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4]   2> 981459 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
> [junit4:junit4]   2> 981459 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
> [junit4:junit4]   2> 981461 T4065 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
> [junit4:junit4]   2> 981461 T4065 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
> [junit4:junit4]   2> 981521 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 981547 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59196
> [junit4:junit4]   2> 981550 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 981550 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 981551 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665
> [junit4:junit4]   2> 981551 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/solr.xml
> [junit4:junit4]   2> 981551 T4065 oasc.CoreContainer.<init> New CoreContainer 911470756
> [junit4:junit4]   2> 981552 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/'
> [junit4:junit4]   2> 981552 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/'
> [junit4:junit4]   2> 981616 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 981616 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 981617 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 981617 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 981617 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 981618 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 981618 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 981619 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 981619 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 981620 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 981624 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 981624 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 981625 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 981626 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 981628 T4085 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@213933ab name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 981628 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 981629 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 981633 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 981634 T4087 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6913f8bf name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 981634 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 981635 T4065 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4:junit4]   2> 981638 T4065 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4:junit4]   2> 981640 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4:junit4]   2> 981641 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59196_rd%2Fu
> [junit4:junit4]   2> 981642 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59196_rd%2Fu
> [junit4:junit4]   2> 981644 T4065 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4:junit4]   2> 981648 T4065 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4:junit4]   2> 981650 T4065 oasc.Overseer.start Overseer (id=89947965583982595-127.0.0.1:59196_rd%2Fu-n_0000000000) starting
> [junit4:junit4]   2> 981652 T4065 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4:junit4]   2> 981654 T4089 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
> [junit4:junit4]   2> 981655 T4065 oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4:junit4]   2> 981656 T4065 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4:junit4]   2> 981657 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 981659 T4088 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4:junit4]   2> 981661 T4090 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 981661 T4090 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 983162 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 983162 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"1",
> [junit4:junit4]   2> 	  "numShards":"1",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"control_collection",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:59196_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:59196/rd/u"}
> [junit4:junit4]   2> 983163 T4088 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
> [junit4:junit4]   2> 983163 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4]   2> 983166 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4]   2> 983662 T4090 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/collection1
> [junit4:junit4]   2> 983662 T4090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
> [junit4:junit4]   2> 983663 T4090 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 983664 T4090 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
> [junit4:junit4]   2> 983665 T4090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/collection1/'
> [junit4:junit4]   2> 983665 T4090 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/collection1/lib/README' to classloader
> [junit4:junit4]   2> 983666 T4090 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 983696 T4090 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 983723 T4090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 983724 T4090 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 983727 T4090 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 983994 T4090 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 983995 T4090 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 983995 T4090 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 984000 T4090 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 984002 T4090 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 984011 T4090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 984013 T4090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 984016 T4090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 984016 T4090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 984017 T4090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 984017 T4090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 984017 T4090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 984018 T4090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 984018 T4090 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 984018 T4090 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372497033665/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/control/data/
> [junit4:junit4]   2> 984018 T4090 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 984018 T4090 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 984019 T4090 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/control/data
> [junit4:junit4]   2> 984019 T4090 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/control/data/index/
> [junit4:junit4]   2> 984019 T4090 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/control/data/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 984020 T4090 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/control/data/index
> [junit4:junit4]   2> 984022 T4090 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/control/data/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 984022 T4090 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 984024 T4090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 984024 T4090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 984024 T4090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 984025 T4090 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 984025 T4090 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 984025 T4090 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 984025 T4090 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 984026 T4090 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 984026 T4090 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 984028 T4090 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 984030 T4090 oass.SolrIndexSearcher.<init> Opening Searcher@79833097 main
> [junit4:junit4]   2> 984031 T4090 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 984031 T4090 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 984034 T4091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79833097 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 984034 T4090 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 984034 T4090 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59196/rd/u collection:control_collection shard:shard1
> [junit4:junit4]   2> 984035 T4090 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
> [junit4:junit4]   2> 984039 T4090 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4]   2> 984040 T4090 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4]   2> 984041 T4090 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4]   2> 984041 T4090 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59196/rd/u/collection1/
> [junit4:junit4]   2> 984041 T4090 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4]   2> 984041 T4090 oasc.SyncStrategy.syncToMe http://127.0.0.1:59196/rd/u/collection1/ has no replicas
> [junit4:junit4]   2> 984041 T4090 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59196/rd/u/collection1/
> [junit4:junit4]   2> 984041 T4090 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
> [junit4:junit4]   2> 984670 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 984676 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4]   2> 984697 T4090 oasc.ZkController.register We are http://127.0.0.1:59196/rd/u/collection1/ and leader is http://127.0.0.1:59196/rd/u/collection1/
> [junit4:junit4]   2> 984697 T4090 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59196/rd/u
> [junit4:junit4]   2> 984697 T4090 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4]   2> 984697 T4090 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4]   2> 984697 T4090 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 984699 T4090 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 984700 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 984700 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 984700 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 984703 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:
> [junit4:junit4]   2> 984704 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 984705 T4094 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3efd2fbe name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 984705 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 984706 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 984707 T4065 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
> [junit4:junit4]   2> 984768 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 984769 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43964
> [junit4:junit4]   2> 984770 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 984770 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 984770 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910
> [junit4:junit4]   2> 984771 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/solr.xml
> [junit4:junit4]   2> 984771 T4065 oasc.CoreContainer.<init> New CoreContainer 860495522
> [junit4:junit4]   2> 984771 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/'
> [junit4:junit4]   2> 984771 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/'
> [junit4:junit4]   2> 984815 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 984816 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 984816 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 984816 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 984817 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 984817 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 984817 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 984817 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 984818 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 984818 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 984820 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 984821 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 984821 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 984821 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 984823 T4105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bfd4d3f name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 984823 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 984824 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 984826 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 984827 T4107 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41692de1 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 984827 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 984829 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 985831 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43964_rd%2Fu
> [junit4:junit4]   2> 985832 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43964_rd%2Fu
> [junit4:junit4]   2> 985834 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
> [junit4:junit4]   2> 985834 T4094 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4]   2> 985834 T4107 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4]   2> 985835 T4087 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4]   2> 985838 T4108 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 985838 T4108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 986180 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 986180 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"1",
> [junit4:junit4]   2> 	  "numShards":"1",
> [junit4:junit4]   2> 	  "shard":"shard1",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"active",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"control_collection",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:59196_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:59196/rd/u"}
> [junit4:junit4]   2> 986182 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"2",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:43964_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43964/rd/u"}
> [junit4:junit4]   2> 986183 T4088 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
> [junit4:junit4]   2> 986183 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4]   2> 986185 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 986185 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 986185 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 986839 T4108 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/collection1
> [junit4:junit4]   2> 986839 T4108 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 986840 T4108 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 986840 T4108 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 986841 T4108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/collection1/'
> [junit4:junit4]   2> 986841 T4108 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/collection1/lib/README' to classloader
> [junit4:junit4]   2> 986841 T4108 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 986861 T4108 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 986893 T4108 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 986894 T4108 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 986897 T4108 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 987174 T4108 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 987175 T4108 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 987175 T4108 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 987180 T4108 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 987182 T4108 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 987192 T4108 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 987194 T4108 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 987196 T4108 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 987197 T4108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 987197 T4108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 987197 T4108 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 987198 T4108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 987198 T4108 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 987198 T4108 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 987198 T4108 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372497036910/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1/
> [junit4:junit4]   2> 987199 T4108 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 987199 T4108 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 987199 T4108 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1
> [junit4:junit4]   2> 987199 T4108 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1/index/
> [junit4:junit4]   2> 987200 T4108 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 987200 T4108 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1/index
> [junit4:junit4]   2> 987202 T4108 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 987203 T4108 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 987204 T4108 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 987204 T4108 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 987205 T4108 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 987205 T4108 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 987206 T4108 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 987206 T4108 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 987206 T4108 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 987206 T4108 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 987207 T4108 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 987209 T4108 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 987211 T4108 oass.SolrIndexSearcher.<init> Opening Searcher@6131f09e main
> [junit4:junit4]   2> 987212 T4108 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 987212 T4108 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 987215 T4109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6131f09e main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 987216 T4108 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 987216 T4108 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43964/rd/u collection:collection1 shard:shard1
> [junit4:junit4]   2> 987217 T4108 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
> [junit4:junit4]   2> 987221 T4108 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4]   2> 987222 T4108 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4]   2> 987222 T4108 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4]   2> 987223 T4108 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43964/rd/u/collection1/
> [junit4:junit4]   2> 987223 T4108 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4]   2> 987223 T4108 oasc.SyncStrategy.syncToMe http://127.0.0.1:43964/rd/u/collection1/ has no replicas
> [junit4:junit4]   2> 987223 T4108 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43964/rd/u/collection1/
> [junit4:junit4]   2> 987223 T4108 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
> [junit4:junit4]   2> 987689 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 987698 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 987698 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 987699 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 987728 T4108 oasc.ZkController.register We are http://127.0.0.1:43964/rd/u/collection1/ and leader is http://127.0.0.1:43964/rd/u/collection1/
> [junit4:junit4]   2> 987728 T4108 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43964/rd/u
> [junit4:junit4]   2> 987728 T4108 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4]   2> 987728 T4108 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4]   2> 987728 T4108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 987730 T4108 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 987731 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 987731 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 987732 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 987829 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 987830 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43124
> [junit4:junit4]   2> 987831 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 987831 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 987832 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937
> [junit4:junit4]   2> 987832 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/solr.xml
> [junit4:junit4]   2> 987832 T4065 oasc.CoreContainer.<init> New CoreContainer 1591643574
> [junit4:junit4]   2> 987832 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/'
> [junit4:junit4]   2> 987833 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/'
> [junit4:junit4]   2> 987868 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 987868 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 987868 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 987868 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 987869 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 987869 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 987869 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 987869 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 987870 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 987870 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 987872 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 987873 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 987873 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 987874 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 987875 T4121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c9faf11 name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 987875 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 987876 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 987878 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 987880 T4123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44e0580d name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 987880 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 987882 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 988884 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43124_rd%2Fu
> [junit4:junit4]   2> 988885 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43124_rd%2Fu
> [junit4:junit4]   2> 988887 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 988887 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4]   2> 988887 T4123 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4]   2> 988887 T4107 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4]   2> 988888 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 988888 T4094 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4]   2> 988888 T4087 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4]   2> 988891 T4124 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 988891 T4124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 989203 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 989203 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"2",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard1",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"active",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:43964_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43964/rd/u"}
> [junit4:junit4]   2> 989206 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"3",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:43124_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43124/rd/u"}
> [junit4:junit4]   2> 989206 T4088 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
> [junit4:junit4]   2> 989206 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4]   2> 989209 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 989209 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 989209 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 989209 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 989892 T4124 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/collection1
> [junit4:junit4]   2> 989893 T4124 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 989893 T4124 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 989893 T4124 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 989894 T4124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/collection1/'
> [junit4:junit4]   2> 989895 T4124 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/collection1/lib/README' to classloader
> [junit4:junit4]   2> 989895 T4124 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 989915 T4124 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 989939 T4124 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 989941 T4124 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 989943 T4124 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 990231 T4124 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 990231 T4124 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 990232 T4124 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 990237 T4124 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 990244 T4124 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 990254 T4124 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 990257 T4124 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 990259 T4124 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 990260 T4124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 990260 T4124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 990260 T4124 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 990261 T4124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 990261 T4124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 990261 T4124 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 990261 T4124 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372497039937/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2/
> [junit4:junit4]   2> 990262 T4124 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 990262 T4124 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 990262 T4124 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2
> [junit4:junit4]   2> 990262 T4124 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2/index/
> [junit4:junit4]   2> 990263 T4124 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 990263 T4124 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2/index
> [junit4:junit4]   2> 990265 T4124 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 990265 T4124 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 990267 T4124 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 990267 T4124 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 990267 T4124 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 990268 T4124 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 990268 T4124 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 990268 T4124 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 990269 T4124 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 990269 T4124 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 990269 T4124 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 990271 T4124 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 990274 T4124 oass.SolrIndexSearcher.<init> Opening Searcher@1166bb57 main
> [junit4:junit4]   2> 990274 T4124 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 990274 T4124 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 990279 T4125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1166bb57 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 990281 T4124 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 990281 T4124 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43124/rd/u collection:collection1 shard:shard2
> [junit4:junit4]   2> 990282 T4124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
> [junit4:junit4]   2> 990286 T4124 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4]   2> 990288 T4124 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4]   2> 990288 T4124 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4]   2> 990289 T4124 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43124/rd/u/collection1/
> [junit4:junit4]   2> 990289 T4124 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4]   2> 990289 T4124 oasc.SyncStrategy.syncToMe http://127.0.0.1:43124/rd/u/collection1/ has no replicas
> [junit4:junit4]   2> 990289 T4124 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43124/rd/u/collection1/
> [junit4:junit4]   2> 990290 T4124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
> [junit4:junit4]   2> 990713 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 990718 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 990718 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 990718 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 990719 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 990744 T4124 oasc.ZkController.register We are http://127.0.0.1:43124/rd/u/collection1/ and leader is http://127.0.0.1:43124/rd/u/collection1/
> [junit4:junit4]   2> 990745 T4124 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43124/rd/u
> [junit4:junit4]   2> 990745 T4124 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4]   2> 990745 T4124 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4]   2> 990745 T4124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 990747 T4124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 990748 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 990748 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 990749 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 990820 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 990822 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60436
> [junit4:junit4]   2> 990823 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 990823 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 990823 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955
> [junit4:junit4]   2> 990823 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/solr.xml
> [junit4:junit4]   2> 990824 T4065 oasc.CoreContainer.<init> New CoreContainer 806534663
> [junit4:junit4]   2> 990824 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/'
> [junit4:junit4]   2> 990824 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/'
> [junit4:junit4]   2> 990864 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 990865 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 990865 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 990865 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 990865 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 990866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 990866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 990866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 990867 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 990867 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 990869 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 990870 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 990870 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 990871 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 990873 T4137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bee2787 name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 990873 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 990874 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 990877 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 990880 T4139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cbb05d0 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 990881 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 990884 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 991886 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60436_rd%2Fu
> [junit4:junit4]   2> 991888 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60436_rd%2Fu
> [junit4:junit4]   2> 991889 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 991889 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 991889 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4]   2> 991889 T4139 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4]   2> 991889 T4107 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4]   2> 991890 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 991891 T4087 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4]   2> 991891 T4123 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4]   2> 991891 T4094 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4]   2> 991893 T4140 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 991894 T4140 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 992223 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 992223 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"3",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard2",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"active",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:43124_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43124/rd/u"}
> [junit4:junit4]   2> 992226 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"4",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:60436_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60436/rd/u"}
> [junit4:junit4]   2> 992226 T4088 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
> [junit4:junit4]   2> 992226 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4]   2> 992229 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 992229 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 992229 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 992229 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 992229 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 992894 T4140 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/collection1
> [junit4:junit4]   2> 992895 T4140 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 992896 T4140 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 992896 T4140 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 992897 T4140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/collection1/'
> [junit4:junit4]   2> 992897 T4140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/collection1/lib/README' to classloader
> [junit4:junit4]   2> 992897 T4140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 992918 T4140 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 992942 T4140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 992943 T4140 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 992946 T4140 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 993218 T4140 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 993218 T4140 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 993219 T4140 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 993223 T4140 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 993225 T4140 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 993235 T4140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 993237 T4140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 993240 T4140 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 993241 T4140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 993241 T4140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 993241 T4140 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 993242 T4140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 993242 T4140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 993242 T4140 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 993242 T4140 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372497042955/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty3/
> [junit4:junit4]   2> 993242 T4140 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 993243 T4140 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 993243 T4140 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty3
> [junit4:junit4]   2> 993243 T4140 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty3/index/
> [junit4:junit4]   2> 993243 T4140 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty3/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 993244 T4140 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty3/index
> [junit4:junit4]   2> 993246 T4140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty3/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 993246 T4140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 993248 T4140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 993248 T4140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 993248 T4140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 993249 T4140 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 993249 T4140 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 993249 T4140 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 993249 T4140 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 993250 T4140 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 993250 T4140 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 993252 T4140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 993255 T4140 oass.SolrIndexSearcher.<init> Opening Searcher@33c9b785 main
> [junit4:junit4]   2> 993255 T4140 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 993255 T4140 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 993259 T4141 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33c9b785 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 993259 T4140 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 993259 T4140 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60436/rd/u collection:collection1 shard:shard1
> [junit4:junit4]   2> 993262 T4140 oasc.ZkController.register We are http://127.0.0.1:60436/rd/u/collection1/ and leader is http://127.0.0.1:43964/rd/u/collection1/
> [junit4:junit4]   2> 993262 T4140 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60436/rd/u
> [junit4:junit4]   2> 993262 T4140 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4]   2> 993262 T4140 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4]   2> ASYNC  NEW_CORE C1151 name=collection1 org.apache.solr.core.SolrCore@76789f3c url=http://127.0.0.1:60436/rd/u/collection1 node=127.0.0.1:60436_rd%2Fu C1151_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:60436_rd%2Fu, base_url=http://127.0.0.1:60436/rd/u}
> [junit4:junit4]   2> 993262 T4142 C1151 P60436 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
> [junit4:junit4]   2> 993262 T4140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 993263 T4142 C1151 P60436 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4]   2> 993263 T4142 C1151 P60436 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4]   2> 993264 T4142 C1151 P60436 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 993264 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 993264 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 993265 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 993265 T4142 C1151 P60436 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 993270 T4103 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4]   2> 993322 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 993324 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55453
> [junit4:junit4]   2> 993324 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 993325 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 993325 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470
> [junit4:junit4]   2> 993325 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/solr.xml
> [junit4:junit4]   2> 993325 T4065 oasc.CoreContainer.<init> New CoreContainer 1166518991
> [junit4:junit4]   2> 993326 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/'
> [junit4:junit4]   2> 993326 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/'
> [junit4:junit4]   2> 993361 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 993362 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 993362 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 993362 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 993362 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 993363 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 993363 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 993363 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 993363 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 993364 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 993366 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 993366 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 993367 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 993367 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 993369 T4154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27e30cc name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 993369 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 993370 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 993372 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 993373 T4156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2662d877 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 993374 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 993376 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 993733 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 993734 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"4",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard1",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"recovering",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:60436_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60436/rd/u"}
> [junit4:junit4]   2> 993737 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 993738 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 993738 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 993738 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 993738 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 993737 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 994270 T4103 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4]   2> 994271 T4103 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:60436_rd%252Fu&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000 
> [junit4:junit4]   2> 994378 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55453_rd%2Fu
> [junit4:junit4]   2> 994379 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55453_rd%2Fu
> [junit4:junit4]   2> 994381 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 994381 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 994381 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 994382 T4156 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4]   2> 994382 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 994381 T4107 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4]   2> 994381 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4]   2> 994383 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 994382 T4094 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4]   2> 994382 T4139 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4]   2> 994383 T4087 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4]   2> 994386 T4157 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 994386 T4157 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 994386 T4123 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4]   2> 995242 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 995243 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"5",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:55453_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:55453/rd/u"}
> [junit4:junit4]   2> 995243 T4088 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
> [junit4:junit4]   2> 995243 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4]   2> 995247 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 995247 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 995247 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 995247 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 995247 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 995247 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 995387 T4157 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/collection1
> [junit4:junit4]   2> 995387 T4157 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 995388 T4157 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 995388 T4157 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 995389 T4157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/collection1/'
> [junit4:junit4]   2> 995390 T4157 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/collection1/lib/README' to classloader
> [junit4:junit4]   2> 995390 T4157 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 995419 T4157 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 995444 T4157 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 995445 T4157 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 995448 T4157 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 995720 T4157 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 995721 T4157 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 995721 T4157 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 995726 T4157 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 995728 T4157 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 995739 T4157 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 995742 T4157 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 995744 T4157 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 995745 T4157 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 995745 T4157 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 995746 T4157 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 995746 T4157 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 995747 T4157 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 995747 T4157 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 995747 T4157 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372497045470/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty4/
> [junit4:junit4]   2> 995747 T4157 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 995748 T4157 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 995748 T4157 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty4
> [junit4:junit4]   2> 995748 T4157 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty4/index/
> [junit4:junit4]   2> 995748 T4157 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty4/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 995749 T4157 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty4/index
> [junit4:junit4]   2> 995751 T4157 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty4/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 995751 T4157 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 995752 T4157 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 995753 T4157 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 995753 T4157 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 995754 T4157 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 995754 T4157 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 995754 T4157 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 995754 T4157 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 995755 T4157 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 995755 T4157 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 995757 T4157 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 995759 T4157 oass.SolrIndexSearcher.<init> Opening Searcher@69f687b3 main
> [junit4:junit4]   2> 995760 T4157 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 995760 T4157 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 995764 T4158 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69f687b3 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 995764 T4157 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 995764 T4157 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55453/rd/u collection:collection1 shard:shard2
> [junit4:junit4]   2> 995766 T4157 oasc.ZkController.register We are http://127.0.0.1:55453/rd/u/collection1/ and leader is http://127.0.0.1:43124/rd/u/collection1/
> [junit4:junit4]   2> 995767 T4157 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55453/rd/u
> [junit4:junit4]   2> 995767 T4157 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4]   2> 995767 T4157 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4]   2> ASYNC  NEW_CORE C1152 name=collection1 org.apache.solr.core.SolrCore@699cddf6 url=http://127.0.0.1:55453/rd/u/collection1 node=127.0.0.1:55453_rd%2Fu C1152_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:55453_rd%2Fu, base_url=http://127.0.0.1:55453/rd/u}
> [junit4:junit4]   2> 995767 T4159 C1152 P55453 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
> [junit4:junit4]   2> 995767 T4157 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 995768 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4]   2> 995768 T4159 C1152 P55453 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4]   2> 995768 T4159 C1152 P55453 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 995769 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 995769 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 995769 T4159 C1152 P55453 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 995769 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 995773 T4116 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4]   2> 995825 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 995827 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54058
> [junit4:junit4]   2> 995827 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 995828 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 995828 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974
> [junit4:junit4]   2> 995828 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/solr.xml
> [junit4:junit4]   2> 995829 T4065 oasc.CoreContainer.<init> New CoreContainer 6363068
> [junit4:junit4]   2> 995829 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/'
> [junit4:junit4]   2> 995829 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/'
> [junit4:junit4]   2> 995865 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 995865 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 995866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 995866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 995866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 995866 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 995867 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 995867 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 995867 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 995867 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 995870 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 995870 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 995870 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 995871 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 995873 T4171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57c2e15a name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 995873 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 995874 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 995876 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 995877 T4173 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ae755d3 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 995878 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 995880 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> ASYNC  NEW_CORE C1153 name=collection1 org.apache.solr.core.SolrCore@76789f3c url=http://127.0.0.1:60436/rd/u/collection1 node=127.0.0.1:60436_rd%2Fu C1153_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:60436_rd%2Fu, base_url=http://127.0.0.1:60436/rd/u}
> [junit4:junit4]   2> 996271 T4142 C1153 P60436 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43964/rd/u/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4]   2> 996272 T4142 C1153 P60436 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60436/rd/u START replicas=[http://127.0.0.1:43964/rd/u/collection1/] nUpdates=100
> [junit4:junit4]   2> 996272 T4142 C1153 P60436 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4]   2> 996272 T4142 C1153 P60436 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4]   2> 996272 T4142 C1153 P60436 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4]   2> 996272 T4142 C1153 P60436 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4]   2> 996272 T4142 C1153 P60436 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4]   2> 996273 T4142 C1153 P60436 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43964/rd/u/collection1/. core=collection1
> [junit4:junit4]   2> 996273 T4142 C1153 P60436 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> ASYNC  NEW_CORE C1154 name=collection1 org.apache.solr.core.SolrCore@4a80b45c url=http://127.0.0.1:43964/rd/u/collection1 node=127.0.0.1:43964_rd%2Fu C1154_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43964_rd%2Fu, base_url=http://127.0.0.1:43964/rd/u, leader=true}
> [junit4:junit4]   2> 996277 T4102 C1154 P43964 oasc.SolrCore.execute [collection1] webapp=/rd/u path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
> [junit4:junit4]   2> 996279 T4101 C1154 P43964 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4]   2> 996281 T4101 C1154 P43964 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 996282 T4101 C1154 P43964 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 996282 T4101 C1154 P43964 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
> [junit4:junit4]   2> 996282 T4101 C1154 P43964 oass.SolrIndexSearcher.<init> Opening Searcher@4a3c187b realtime
> [junit4:junit4]   2> 996283 T4101 C1154 P43964 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4]   2> 996283 T4101 C1154 P43964 oasup.LogUpdateProcessor.finish [collection1] webapp=/rd/u path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
> [junit4:junit4]   2> 996284 T4142 C1153 P60436 oash.ReplicationHandler.inform Commits will be reserved for  10000
> [junit4:junit4]   2> 996284 T4142 C1153 P60436 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4]   2> 996285 T4100 C1154 P43964 oash.ReplicationHandler.inform Commits will be reserved for  10000
> [junit4:junit4]   2> 996286 T4100 C1154 P43964 oasc.SolrCore.execute [collection1] webapp=/rd/u path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
> [junit4:junit4]   2> 996286 T4142 C1153 P60436 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4]   2> 996286 T4142 C1153 P60436 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4]   2> 996286 T4142 C1153 P60436 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4]   2> 996287 T4142 C1153 P60436 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 996288 T4142 C1153 P60436 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4]   2> 996752 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 996752 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"5",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard2",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"recovering",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:55453_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:55453/rd/u"}
> [junit4:junit4]   2> 996755 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"4",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard1",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"active",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:60436_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60436/rd/u"}
> [junit4:junit4]   2> 996758 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996758 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996758 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996758 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996758 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996758 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996758 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996773 T4116 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4]   2> 996774 T4116 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:55453_rd%252Fu&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
> [junit4:junit4]   2> 996882 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54058_rd%2Fu
> [junit4:junit4]   2> 996884 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54058_rd%2Fu
> [junit4:junit4]   2> 996885 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996885 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996885 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996885 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996885 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4]   2> 996886 T4107 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996886 T4156 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996886 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 996887 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 996888 T4094 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996888 T4139 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996888 T4087 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996888 T4123 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996889 T4173 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4]   2> 996892 T4175 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 996893 T4175 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 998264 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 998264 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"6",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:54058_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:54058/rd/u"}
> [junit4:junit4]   2> 998265 T4088 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
> [junit4:junit4]   2> 998265 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4]   2> 998268 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 998268 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 998268 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 998268 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 998268 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 998268 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 998268 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2>  C1152_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:55453_rd%2Fu, base_url=http://127.0.0.1:55453/rd/u}
> [junit4:junit4]   2> 998775 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43124/rd/u/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4]   2> 998775 T4159 C1152 P55453 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55453/rd/u START replicas=[http://127.0.0.1:43124/rd/u/collection1/] nUpdates=100
> [junit4:junit4]   2> 998775 T4159 C1152 P55453 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4]   2> 998776 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4]   2> 998776 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4]   2> 998776 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4]   2> 998776 T4159 C1152 P55453 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4]   2> 998776 T4159 C1152 P55453 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43124/rd/u/collection1/. core=collection1
> [junit4:junit4]   2> 998777 T4159 C1152 P55453 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> ASYNC  NEW_CORE C1155 name=collection1 org.apache.solr.core.SolrCore@609724b8 url=http://127.0.0.1:43124/rd/u/collection1 node=127.0.0.1:43124_rd%2Fu C1155_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43124_rd%2Fu, base_url=http://127.0.0.1:43124/rd/u, leader=true}
> [junit4:junit4]   2> 998780 T4118 C1155 P43124 oasc.SolrCore.execute [collection1] webapp=/rd/u path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
> [junit4:junit4]   2> 998783 T4119 C1155 P43124 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4]   2> 998785 T4119 C1155 P43124 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 998785 T4119 C1155 P43124 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 998786 T4119 C1155 P43124 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
> [junit4:junit4]   2> 998786 T4119 C1155 P43124 oass.SolrIndexSearcher.<init> Opening Searcher@5dd9c621 realtime
> [junit4:junit4]   2> 998786 T4119 C1155 P43124 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4]   2> 998787 T4119 C1155 P43124 oasup.LogUpdateProcessor.finish [collection1] webapp=/rd/u path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
> [junit4:junit4]   2> 998787 T4159 C1152 P55453 oash.ReplicationHandler.inform Commits will be reserved for  10000
> [junit4:junit4]   2> 998788 T4159 C1152 P55453 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4]   2> 998789 T4116 C1155 P43124 oash.ReplicationHandler.inform Commits will be reserved for  10000
> [junit4:junit4]   2> 998789 T4116 C1155 P43124 oasc.SolrCore.execute [collection1] webapp=/rd/u path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
> [junit4:junit4]   2> 998790 T4159 C1152 P55453 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4]   2> 998790 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4]   2> 998790 T4159 C1152 P55453 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4]   2> 998790 T4159 C1152 P55453 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 998791 T4159 C1152 P55453 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4]   2> 998894 T4175 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/collection1
> [junit4:junit4]   2> 998894 T4175 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 998895 T4175 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 998895 T4175 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 998897 T4175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/collection1/'
> [junit4:junit4]   2> 998897 T4175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/collection1/lib/README' to classloader
> [junit4:junit4]   2> 998897 T4175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 998919 T4175 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 998944 T4175 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 998945 T4175 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 998948 T4175 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 999235 T4175 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 999236 T4175 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 999236 T4175 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 999242 T4175 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 999244 T4175 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 999253 T4175 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 999256 T4175 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 999258 T4175 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 999259 T4175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 999259 T4175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 999259 T4175 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 999260 T4175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 999260 T4175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 999260 T4175 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 999261 T4175 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372497047974/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty5/
> [junit4:junit4]   2> 999261 T4175 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 999261 T4175 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 999262 T4175 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty5
> [junit4:junit4]   2> 999262 T4175 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty5/index/
> [junit4:junit4]   2> 999262 T4175 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty5/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 999262 T4175 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty5/index
> [junit4:junit4]   2> 999265 T4175 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty5/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 999265 T4175 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 999266 T4175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 999266 T4175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 999267 T4175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 999267 T4175 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 999268 T4175 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 999268 T4175 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 999269 T4175 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 999270 T4175 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 999270 T4175 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 999272 T4175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 999274 T4175 oass.SolrIndexSearcher.<init> Opening Searcher@67c98642 main
> [junit4:junit4]   2> 999275 T4175 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 999275 T4175 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 999278 T4177 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67c98642 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 999279 T4175 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 999280 T4175 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54058/rd/u collection:collection1 shard:shard1
> [junit4:junit4]   2> 999282 T4175 oasc.ZkController.register We are http://127.0.0.1:54058/rd/u/collection1/ and leader is http://127.0.0.1:43964/rd/u/collection1/
> [junit4:junit4]   2> 999282 T4175 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54058/rd/u
> [junit4:junit4]   2> 999282 T4175 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4]   2> 999282 T4175 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4]   2> ASYNC  NEW_CORE C1156 name=collection1 org.apache.solr.core.SolrCore@3c1386ad url=http://127.0.0.1:54058/rd/u/collection1 node=127.0.0.1:54058_rd%2Fu C1156_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:54058_rd%2Fu, base_url=http://127.0.0.1:54058/rd/u}
> [junit4:junit4]   2> 999283 T4178 C1156 P54058 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
> [junit4:junit4]   2> 999283 T4175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 999284 T4178 C1156 P54058 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4]   2> 999284 T4178 C1156 P54058 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4]   2> 999285 T4178 C1156 P54058 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 999285 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 999285 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 999286 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 999287 T4178 C1156 P54058 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 999291 T4103 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4]   2> 999343 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 999344 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33104
> [junit4:junit4]   2> 999345 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 999345 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 999346 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490
> [junit4:junit4]   2> 999346 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/solr.xml
> [junit4:junit4]   2> 999346 T4065 oasc.CoreContainer.<init> New CoreContainer 1408819424
> [junit4:junit4]   2> 999346 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/'
> [junit4:junit4]   2> 999347 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/'
> [junit4:junit4]   2> 999383 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 999383 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 999383 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 999384 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 999384 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 999384 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 999384 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 999385 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 999385 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 999385 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 999388 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 999388 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 999388 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 999389 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 999391 T4190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49177f4e name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 999391 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 999392 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 999402 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 999404 T4192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5445cc38 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 999404 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 999406 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> 999772 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 999773 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"5",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard2",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"active",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:55453_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:55453/rd/u"}
> [junit4:junit4]   2> 999776 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"6",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard1",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"recovering",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:54058_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:54058/rd/u"}
> [junit4:junit4]   2> 999780 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 999780 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000292 T4103 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4]   2> 1000292 T4103 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:54058_rd%252Fu&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
> [junit4:junit4]   2> 1000409 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33104_rd%2Fu
> [junit4:junit4]   2> 1000410 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33104_rd%2Fu
> [junit4:junit4]   2> 1000412 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000412 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000412 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000412 T4156 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000412 T4107 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000412 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000412 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000412 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4]   2> 1000413 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1000412 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1000414 T4087 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000414 T4192 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000414 T4123 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000415 T4139 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000415 T4173 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000415 T4094 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4]   2> 1000418 T4193 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 1000418 T4193 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 1001285 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 1001286 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"7",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:33104_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:33104/rd/u"}
> [junit4:junit4]   2> 1001286 T4088 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
> [junit4:junit4]   2> 1001286 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4]   2> 1001291 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001291 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1001419 T4193 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/collection1
> [junit4:junit4]   2> 1001419 T4193 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 1001420 T4193 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 1001420 T4193 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 1001422 T4193 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/collection1/'
> [junit4:junit4]   2> 1001422 T4193 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/collection1/lib/README' to classloader
> [junit4:junit4]   2> 1001422 T4193 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 1001444 T4193 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 1001469 T4193 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 1001470 T4193 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 1001473 T4193 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 1001741 T4193 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 1001741 T4193 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 1001742 T4193 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 1001747 T4193 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 1001749 T4193 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 1001758 T4193 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 1001760 T4193 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 1001763 T4193 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 1001763 T4193 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 1001763 T4193 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 1001764 T4193 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 1001764 T4193 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 1001765 T4193 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 1001765 T4193 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 1001765 T4193 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372497051490/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty6/
> [junit4:junit4]   2> 1001765 T4193 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 1001765 T4193 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 1001766 T4193 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty6
> [junit4:junit4]   2> 1001766 T4193 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty6/index/
> [junit4:junit4]   2> 1001766 T4193 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty6/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 1001766 T4193 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty6/index
> [junit4:junit4]   2> 1001769 T4193 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty6/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 1001769 T4193 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 1001771 T4193 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 1001771 T4193 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 1001771 T4193 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 1001772 T4193 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 1001772 T4193 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 1001772 T4193 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 1001772 T4193 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 1001773 T4193 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 1001773 T4193 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 1001775 T4193 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 1001778 T4193 oass.SolrIndexSearcher.<init> Opening Searcher@2e71bef6 main
> [junit4:junit4]   2> 1001778 T4193 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 1001778 T4193 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 1001782 T4194 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e71bef6 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 1001783 T4193 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 1001783 T4193 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33104/rd/u collection:collection1 shard:shard2
> [junit4:junit4]   2> 1001786 T4193 oasc.ZkController.register We are http://127.0.0.1:33104/rd/u/collection1/ and leader is http://127.0.0.1:43124/rd/u/collection1/
> [junit4:junit4]   2> 1001786 T4193 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33104/rd/u
> [junit4:junit4]   2> 1001786 T4193 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4]   2> 1001786 T4193 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4]   2> ASYNC  NEW_CORE C1157 name=collection1 org.apache.solr.core.SolrCore@1c1a1ad4 url=http://127.0.0.1:33104/rd/u/collection1 node=127.0.0.1:33104_rd%2Fu C1157_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:33104_rd%2Fu, base_url=http://127.0.0.1:33104/rd/u}
> [junit4:junit4]   2> 1001787 T4195 C1157 P33104 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
> [junit4:junit4]   2> 1001787 T4193 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 1001787 T4195 C1157 P33104 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4]   2> 1001788 T4195 C1157 P33104 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4]   2> 1001788 T4195 C1157 P33104 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 1001788 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 1001789 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 1001789 T4195 C1157 P33104 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 1001789 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 1001793 T4118 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4]   2> 1001845 T4065 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4]   2> 1001847 T4065 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41890
> [junit4:junit4]   2> 1001847 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4]   2> 1001848 T4065 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4]   2> 1001848 T4065 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994
> [junit4:junit4]   2> 1001848 T4065 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/solr.xml
> [junit4:junit4]   2> 1001848 T4065 oasc.CoreContainer.<init> New CoreContainer 1584307977
> [junit4:junit4]   2> 1001849 T4065 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/'
> [junit4:junit4]   2> 1001849 T4065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/'
> [junit4:junit4]   2> 1001891 T4065 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4]   2> 1001892 T4065 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4]   2> 1001892 T4065 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4]   2> 1001892 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4]   2> 1001893 T4065 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4]   2> 1001893 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4]   2> 1001893 T4065 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4]   2> 1001893 T4065 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4]   2> 1001894 T4065 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4]   2> 1001894 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4]   2> 1001896 T4065 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4]   2> 1001897 T4065 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59208/solr
> [junit4:junit4]   2> 1001897 T4065 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4]   2> 1001898 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 1001899 T4207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fe039aa name:ZooKeeperConnection Watcher:127.0.0.1:59208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 1001900 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 1001901 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4]   2> 1001903 T4065 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4]   2> 1001904 T4209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@205776a0 name:ZooKeeperConnection Watcher:127.0.0.1:59208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4]   2> 1001904 T4065 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4]   2> 1001907 T4065 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
> [junit4:junit4]   2> ASYNC  NEW_CORE C1158 name=collection1 org.apache.solr.core.SolrCore@3c1386ad url=http://127.0.0.1:54058/rd/u/collection1 node=127.0.0.1:54058_rd%2Fu C1158_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:54058_rd%2Fu, base_url=http://127.0.0.1:54058/rd/u}
> [junit4:junit4]   2> 1002293 T4178 C1158 P54058 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43964/rd/u/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4]   2> 1002293 T4178 C1158 P54058 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54058/rd/u START replicas=[http://127.0.0.1:43964/rd/u/collection1/] nUpdates=100
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43964/rd/u/collection1/. core=collection1
> [junit4:junit4]   2> 1002294 T4178 C1158 P54058 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> ASYNC  NEW_CORE C1159 name=collection1 org.apache.solr.core.SolrCore@4a80b45c url=http://127.0.0.1:43964/rd/u/collection1 node=127.0.0.1:43964_rd%2Fu C1159_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43964_rd%2Fu, base_url=http://127.0.0.1:43964/rd/u, leader=true}
> [junit4:junit4]   2> 1002296 T4103 C1159 P43964 oasc.SolrCore.execute [collection1] webapp=/rd/u path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
> [junit4:junit4]   2> 1002297 T4100 C1159 P43964 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4]   2> 1002298 T4100 C1159 P43964 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
> [junit4:junit4]   2> 1002298 T4100 C1159 P43964 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4]   2> 1002298 T4100 C1159 P43964 oasup.LogUpdateProcessor.finish [collection1] webapp=/rd/u path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
> [junit4:junit4]   2> 1002299 T4178 C1158 P54058 oash.ReplicationHandler.inform Commits will be reserved for  10000
> [junit4:junit4]   2> 1002299 T4178 C1158 P54058 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4]   2> 1002301 T4101 C1159 P43964 oasc.SolrCore.execute [collection1] webapp=/rd/u path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
> [junit4:junit4]   2> 1002301 T4178 C1158 P54058 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4]   2> 1002301 T4178 C1158 P54058 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4]   2> 1002302 T4178 C1158 P54058 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4]   2> 1002302 T4178 C1158 P54058 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 1002303 T4178 C1158 P54058 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4]   2> 1002795 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 1002796 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"7",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard2",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"recovering",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:33104_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:33104/rd/u"}
> [junit4:junit4]   2> 1002798 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"6",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":"shard1",
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"active",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:54058_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:54058/rd/u"}
> [junit4:junit4]   2> 1002800 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002800 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002909 T4065 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41890_rd%2Fu
> [junit4:junit4]   2> 1002910 T4065 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41890_rd%2Fu
> [junit4:junit4]   2> 1002912 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002912 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002912 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002913 T4156 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002913 T4209 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002913 T4209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1002912 T4107 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002912 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002912 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002914 T4173 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002914 T4139 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002912 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4]   2> 1002914 T4087 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002913 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1002913 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1002915 T4094 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002915 T4123 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002916 T4192 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4]   2> 1002918 T4211 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4]   2> 1002918 T4211 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 1003793 T4118 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
> [junit4:junit4]   2> 1003794 T4118 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:33104_rd%252Fu&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001 
> [junit4:junit4]   2> 1004305 T4088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 1004306 T4088 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
> [junit4:junit4]   2> 	  "operation":"state",
> [junit4:junit4]   2> 	  "core_node_name":"8",
> [junit4:junit4]   2> 	  "numShards":"2",
> [junit4:junit4]   2> 	  "shard":null,
> [junit4:junit4]   2> 	  "roles":null,
> [junit4:junit4]   2> 	  "state":"down",
> [junit4:junit4]   2> 	  "shard_state":"active",
> [junit4:junit4]   2> 	  "core":"collection1",
> [junit4:junit4]   2> 	  "collection":"collection1",
> [junit4:junit4]   2> 	  "shard_range":null,
> [junit4:junit4]   2> 	  "node_name":"127.0.0.1:41890_rd%2Fu",
> [junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41890/rd/u"}
> [junit4:junit4]   2> 1004306 T4088 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
> [junit4:junit4]   2> 1004306 T4088 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4]   2> 1004309 T4087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4173 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004309 T4156 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4]   2> 1004919 T4211 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/collection1
> [junit4:junit4]   2> 1004919 T4211 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4]   2> 1004920 T4211 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4]   2> 1004920 T4211 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4]   2> 1004921 T4211 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/collection1/'
> [junit4:junit4]   2> 1004922 T4211 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/collection1/lib/README' to classloader
> [junit4:junit4]   2> 1004922 T4211 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/collection1/lib/classes/' to classloader
> [junit4:junit4]   2> 1004943 T4211 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4]   2> 1004968 T4211 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4]   2> 1004969 T4211 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4]   2> 1004972 T4211 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4]   2> 1005245 T4211 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 1005245 T4211 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4]   2> 1005245 T4211 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4]   2> 1005250 T4211 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4]   2> 1005252 T4211 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4]   2> 1005260 T4211 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 1005263 T4211 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4]   2> 1005265 T4211 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 1005266 T4211 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 1005266 T4211 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 1005266 T4211 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4]   2> 1005267 T4211 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4]   2> 1005267 T4211 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4]   2> 1005267 T4211 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 1005267 T4211 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372497053994/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty7/
> [junit4:junit4]   2> 1005268 T4211 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4d2b18
> [junit4:junit4]   2> 1005268 T4211 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4]   2> 1005268 T4211 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty7
> [junit4:junit4]   2> 1005268 T4211 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty7/index/
> [junit4:junit4]   2> 1005269 T4211 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty7/index' doesn't exist. Creating new index...
> [junit4:junit4]   2> 1005269 T4211 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty7/index
> [junit4:junit4]   2> 1005271 T4211 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372497033445/jetty7/index,segFN=segments_1,generation=1}
> [junit4:junit4]   2> 1005272 T4211 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4]   2> 1005273 T4211 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4]   2> 1005273 T4211 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4]   2> 1005274 T4211 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4]   2> 1005274 T4211 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4]   2> 1005274 T4211 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4]   2> 1005275 T4211 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4]   2> 1005275 T4211 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4]   2> 1005275 T4211 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4]   2> 1005275 T4211 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4]   2> 1005277 T4211 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4]   2> 1005280 T4211 oass.SolrIndexSearcher.<init> Opening Searcher@7bd8d56e main
> [junit4:junit4]   2> 1005280 T4211 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4]   2> 1005280 T4211 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4]   2> 1005283 T4212 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7bd8d56e main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4]   2> 1005284 T4211 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4]   2> 1005284 T4211 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41890/rd/u collection:collection1 shard:shard1
> [junit4:junit4]   2> 1005286 T4211 oasc.ZkController.register We are http://127.0.0.1:41890/rd/u/collection1/ and leader is http://127.0.0.1:43964/rd/u/collection1/
> [junit4:junit4]   2> 1005287 T4211 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41890/rd/u
> [junit4:junit4]   2> 1005287 T4211 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4]   2> 1005287 T4211 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4]   2> ASYNC  NEW_CORE C1160 name=collection1 org.apache.solr.core.SolrCore@411393fa url=http://127.0.0.1:41890/rd/u/collection1 node=127.0.0.1:41890_rd%2Fu C1160_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41890_rd%2Fu, base_url=http://127.0.0.1:41890/rd/u}
> [junit4:junit4]   2> 1005287 T4213 C1160 P41890 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
> [junit4:junit4]   2> 1005288 T4213 C1160 P41890 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4]   2> 1005288 T4211 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 1005288 T4213 C1160 P41890 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4]   2> 1005289 T4213 C1160 P41890 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4]   2> 1005289 T4065 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4]   2> 1005290 T4065 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4]   2> 1005290 T4213 C1160 P41890 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 1005290 T4065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4]   2> 1005294 T4065 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> 1005295 T4103 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 8, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4]   2> 1005295 T4065 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
> [junit4:junit4]   2> 1005296 T4065 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
> [junit4:junit4]   2> ASYNC  NEW_CORE C1161 name=collection1 org.apache.solr.core.SolrCore
> 
> [...truncated too long message...]
> 
> ollections/collection1/leaders (2)
> [junit4:junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
> [junit4:junit4]   1>      DATA:
> [junit4:junit4]   1>          {
> [junit4:junit4]   1>            "core":"collection1",
> [junit4:junit4]   1>            "node_name":"127.0.0.1:60436_rd%2Fu",
> [junit4:junit4]   1>            "base_url":"http://127.0.0.1:60436/rd/u"}
> [junit4:junit4]   1>      /solr/collections/collection1/leaders/shard2 (0)
> [junit4:junit4]   1>      DATA:
> [junit4:junit4]   1>          {
> [junit4:junit4]   1>            "core":"collection1",
> [junit4:junit4]   1>            "node_name":"127.0.0.1:55453_rd%2Fu",
> [junit4:junit4]   1>            "base_url":"http://127.0.0.1:55453/rd/u"}
> [junit4:junit4]   1>    /solr/collections/control_collection (3)
> [junit4:junit4]   1>    DATA:
> [junit4:junit4]   1>        {"configName":"conf1"}
> [junit4:junit4]   1>     /solr/collections/control_collection/shards (0)
> [junit4:junit4]   1>     /solr/collections/control_collection/leader_elect (1)
> [junit4:junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
> [junit4:junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
> [junit4:junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/89947965583982595-1-n_0000000000 (0)
> [junit4:junit4]   1>     /solr/collections/control_collection/leaders (1)
> [junit4:junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
> [junit4:junit4]   1>      DATA:
> [junit4:junit4]   1>          {
> [junit4:junit4]   1>            "core":"collection1",
> [junit4:junit4]   1>            "node_name":"127.0.0.1:59196_rd%2Fu",
> [junit4:junit4]   1>            "base_url":"http://127.0.0.1:59196/rd/u"}
> [junit4:junit4]   1>   /solr/overseer_elect (2)
> [junit4:junit4]   1>    /solr/overseer_elect/election (6)
> [junit4:junit4]   1>     /solr/overseer_elect/election/89947965583982595-127.0.0.1:59196_rd%2Fu-n_0000000000 (0)
> [junit4:junit4]   1>     /solr/overseer_elect/election/89947965583982604-127.0.0.1:55453_rd%2Fu-n_0000000004 (0)
> [junit4:junit4]   1>     /solr/overseer_elect/election/89947965583982606-127.0.0.1:54058_rd%2Fu-n_0000000005 (0)
> [junit4:junit4]   1>     /solr/overseer_elect/election/89947965583982610-127.0.0.1:41890_rd%2Fu-n_0000000007 (0)
> [junit4:junit4]   1>     /solr/overseer_elect/election/89947965583982602-127.0.0.1:60436_rd%2Fu-n_0000000003 (0)
> [junit4:junit4]   1>     /solr/overseer_elect/election/89947965583982612-127.0.0.1:43964_rd%2Fu-n_0000000008 (0)
> [junit4:junit4]   1>    /solr/overseer_elect/leader (0)
> [junit4:junit4]   1>    DATA:
> [junit4:junit4]   1>        {"id":"89947965583982595-127.0.0.1:59196_rd%2Fu-n_0000000000"}
> [junit4:junit4]   1>  /zookeeper (1)
> [junit4:junit4]   1>  DATA:
> [junit4:junit4]   1>      
> [junit4:junit4]   1> 
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=72DE2FB2FAE961FB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_DZ -Dtests.timezone=Asia/Oral -Dtests.file.encoding=UTF-8
> [junit4:junit4] FAILURE  105s J0 | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: document count mismatch.  control=99 sum(shards)=55 cloudClient=99
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([72DE2FB2FAE961FB:F338A1AA8DB601C7]:0)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1147)
> [junit4:junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:194)
> [junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
> [junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4]   2> 1086146 T4065 oas.SolrTestCaseJ4.deleteCore ###deleteCore
> [junit4:junit4]   2> 104912 T4064 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
> [junit4:junit4]   2> 1087326 T4209 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4]   2> 1087327 T4209 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
> [junit4:junit4]   2> 1087327 T4209 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {timestamp=Pulsing41(freqCutoff=14 minBlockSize=63 maxBlockSize=128), range_facet_l=Pulsing41(freqCutoff=14 minBlockSize=63 maxBlockSize=128), id=PostingsFormat(name=SimpleText), range_facet_sl=PostingsFormat(name=SimpleText), a_t=Pulsing41(freqCutoff=14 minBlockSize=63 maxBlockSize=128), text=PostingsFormat(name=Direct), range_facet_si=Pulsing41(freqCutoff=14 minBlockSize=63 maxBlockSize=128), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=Pulsing41(freqCutoff=14 minBlockSize=63 maxBlockSize=128), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), intDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_DZ, timezone=Asia/Oral
> [junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=5,free=46200776,total=331087872
> [junit4:junit4]   2> NOTE: All tests run in this JVM: [TestConfig, TestWriterPerf, TestFieldResource, TestSweetSpotSimilarityFactory, LukeRequestHandlerTest, DocValuesTest, SystemInfoHandlerTest, DocumentBuilderTest, TestSolrQueryParserDefaultOperatorResource, BasicDistributedZkTest, DateFieldTest, AnalysisAfterCoreReloadTest, SuggesterTest, TestJoin, ScriptEngineTest, RAMDirectoryFactoryTest, PolyFieldTest, TestLMDirichletSimilarityFactory, TestReversedWildcardFilterFactory, TestDefaultSearchFieldResource, TestPseudoReturnFields, TestPerFieldSimilarity, ParsingFieldUpdateProcessorsTest, CoreAdminHandlerTest, TestDynamicFieldCollectionResource, ResourceLoaderTest, BasicFunctionalityTest, ZkCLITest, DistributedSpellCheckComponentTest, UpdateRequestProcessorFactoryTest, TermVectorComponentTest, TermsComponentTest, TestFieldTypeCollectionResource, SolrTestCaseJ4Test, CacheHeaderTest, TestSort, UnloadDistributedZkTest, TestStressVersions, JsonLoaderTest, PreAnalyzedFieldTest, SoftAutoCommitTest, TestStressRecovery, TestFuzzyAnalyzedSuggestions, TestXIncludeConfig, SliceStateUpdateTest, LegacyHTMLStripCharFilterTest, SuggesterFSTTest, DirectUpdateHandlerOptimizeTest, TestDocumentBuilder, RegexBoostProcessorTest, SearchHandlerTest, TestExtendedDismaxParser, TestSolrIndexConfig, TestFastLRUCache, SyncSliceTest, TestSolrQueryParser, StatelessScriptUpdateProcessorFactoryTest, OverseerTest, SOLR749Test, MinimalSchemaTest, TestPluginEnable, ExternalFileFieldSortTest, MBeansHandlerTest, PluginInfoTest, SolrCoreTest, TestPHPSerializedResponseWriter, TestSolrQueryParserResource, TestIBSimilarityFactory, MoreLikeThisHandlerTest, TestMaxScoreQueryParser, ClusterStateUpdateTest, BasicDistributedZk2Test, ShardRoutingTest, TestReplicationHandler, TestZkChroot, ShardRoutingCustomTest, AutoCommitTest, TestFaceting, TestRecovery, TestHashPartitioner, ZkControllerTest, TestRealTimeGet, TestMultiCoreConfBootstrap, TestReload, TestCoreContainer, StatsComponentTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestFunctionQuery, DirectUpdateHandlerTest, TestBadConfig, OverseerCollectionProcessorTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, TestCoreDiscovery, SpellCheckCollatorTest, TestTrie, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, TestRemoteStreaming, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, TestValueSourceCache, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, IndexSchemaTest, ReturnFieldsTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, TestLFUCache, HighlighterConfigTest, TestQuerySenderListener, UpdateParamsTest, ResponseLogComponentTest, BadComponentTest, TestSolrDeletionPolicy2, SampleTest, TestBinaryField, TestSearchPerf, TestPostingsSolrHighlighter, TestCharFilters, SynonymTokenizerTest, TestBM25SimilarityFactory, TimeZoneUtilsTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest]
> [junit4:junit4] Completed on J0 in 106.19s, 1 test, 1 failure <<< FAILURES!
> 
> [...truncated 62 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1248: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:892: There were test failures: 299 suites, 1290 tests, 1 failure, 12 ignored (6 assumptions)
> 
> Total time: 38 minutes 35 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message