lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1686 - Still Failing!
Date Thu, 03 Jul 2014 07:53:22 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1686/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=72 sum(shards)=73 cloudClient=73

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=72 sum(shards)=73 cloudClient=73
	at __randomizedtesting.SeedInfo.seed([D2E263878DC21E9D:5304ED9FFA9D7EA1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1209)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:226)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11281 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/init-core-data-001
   [junit4]   2> 1435401 T5549 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /b_u/vk
   [junit4]   2> 1435408 T5549 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1435409 T5549 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1435410 T5550 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1435511 T5549 oasc.ZkTestServer.run start zk server on port:50861
   [junit4]   2> 1435515 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1435521 T5556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3806fd9a name:ZooKeeperConnection Watcher:127.0.0.1:50861 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1435522 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1435522 T5549 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1435532 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1435536 T5558 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@304c3a10 name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1435536 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1435537 T5549 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1435544 T5549 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1435551 T5549 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1435557 T5549 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1435565 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1435566 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1435578 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1435580 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1435588 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1435589 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1435595 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1435596 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1435605 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1435606 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1435612 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1435613 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1435622 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1435622 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1435631 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1435632 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1435641 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1435642 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1435653 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1435654 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1435662 T5549 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1435663 T5549 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1437369 T5549 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1437396 T5549 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50864
   [junit4]   2> 1437397 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1437397 T5549 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1437398 T5549 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002
   [junit4]   2> 1437398 T5549 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/'
   [junit4]   2> 1437440 T5549 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/solr.xml
   [junit4]   2> 1437481 T5549 oasc.CoreContainer.<init> New CoreContainer 1036647488
   [junit4]   2> 1437482 T5549 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/]
   [junit4]   2> 1437483 T5549 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1437483 T5549 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1437484 T5549 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1437484 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1437485 T5549 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1437485 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1437485 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1437486 T5549 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1437486 T5549 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1437487 T5549 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1437487 T5549 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1437488 T5549 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1437488 T5549 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50861/solr
   [junit4]   2> 1437488 T5549 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1437490 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1437495 T5569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16ef7930 name:ZooKeeperConnection Watcher:127.0.0.1:50861 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1437496 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1437501 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1437504 T5571 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20f2cabe name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1437504 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1437508 T5549 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1437518 T5549 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1437529 T5549 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1437540 T5549 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1437551 T5549 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1437562 T5549 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1437567 T5549 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50864_b_u%2Fvk
   [junit4]   2> 1437569 T5549 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50864_b_u%2Fvk
   [junit4]   2> 1437576 T5549 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1437581 T5549 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1437590 T5549 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50864_b_u%2Fvk
   [junit4]   2> 1437590 T5549 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1437596 T5549 oasc.Overseer.start Overseer (id=92036883248775171-127.0.0.1:50864_b_u%2Fvk-n_0000000000) starting
   [junit4]   2> 1437605 T5549 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1437641 T5573 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1437643 T5549 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1437648 T5549 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1437653 T5549 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1437665 T5572 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1437671 T5574 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1437671 T5574 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1437675 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1437675 T5574 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1437688 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50864/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50864_b_u%2Fvk",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1437689 T5572 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1437689 T5572 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 1437689 T5572 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1437696 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1437698 T5571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1438677 T5574 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1438678 T5574 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1438681 T5574 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1438681 T5574 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1438687 T5574 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1438688 T5574 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/collection1/'
   [junit4]   2> 1438692 T5574 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1438693 T5574 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1438694 T5574 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1438747 T5574 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1438759 T5574 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1438763 T5574 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1438774 T5574 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1438875 T5574 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1438876 T5574 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1438877 T5574 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1438886 T5574 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1439007 T5574 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1439008 T5574 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1439009 T5574 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-002/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data/
   [junit4]   2> 1439009 T5574 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@600bc100
   [junit4]   2> 1439014 T5574 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data
   [junit4]   2> 1439015 T5574 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data/index/
   [junit4]   2> 1439018 T5574 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1439019 T5574 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data/index
   [junit4]   2> 1439021 T5574 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=83.3212890625, floorSegmentMB=1.5, forceMergeDeletesPctAllowed=3.1730657031306997, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1439033 T5574 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1439033 T5574 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1439038 T5574 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1439038 T5574 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1439039 T5574 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1439039 T5574 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1439039 T5574 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1439039 T5574 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1439040 T5574 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1439041 T5574 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1439041 T5574 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1439041 T5574 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1439042 T5574 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1439042 T5574 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1439042 T5574 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1439043 T5574 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1439043 T5574 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1439044 T5574 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1439044 T5574 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1439053 T5574 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1439055 T5574 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1439056 T5574 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1439056 T5574 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5752913056219086]
   [junit4]   2> 1439058 T5574 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1439058 T5574 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1439058 T5574 oass.SolrIndexSearcher.<init> Opening Searcher@4011a163[collection1] main
   [junit4]   2> 1439059 T5574 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1439061 T5574 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1439061 T5574 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1439062 T5574 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1439063 T5574 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1439063 T5574 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1439064 T5574 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1439064 T5574 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1439066 T5574 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1439070 T5574 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1439071 T5574 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1439071 T5574 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1439073 T5575 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4011a163[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1439074 T5574 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1439076 T5578 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50864/b_u/vk collection:control_collection shard:shard1
   [junit4]   2> 1439077 T5549 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1439078 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1439078 T5578 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1439079 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1439083 T5580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b20237b name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1439084 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1439085 T5549 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1439093 T5578 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1439094 T5549 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1439094 T5549 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1439096 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1439099 T5582 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1db1ab9b name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1439100 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1439100 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439102 T5578 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1439102 T5578 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1114 name=collection1 org.apache.solr.core.SolrCore@2b2744a3 url=http://127.0.0.1:50864/b_u/vk/collection1 node=127.0.0.1:50864_b_u%2Fvk C1114_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:50864/b_u/vk, core=collection1, node_name=127.0.0.1:50864_b_u%2Fvk}
   [junit4]   2> 1439102 T5578 C1114 P50864 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50864/b_u/vk/collection1/
   [junit4]   2> 1439103 T5578 C1114 P50864 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1439103 T5578 C1114 P50864 oasc.SyncStrategy.syncToMe http://127.0.0.1:50864/b_u/vk/collection1/ has no replicas
   [junit4]   2> 1439103 T5578 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50864/b_u/vk/collection1/ shard1
   [junit4]   2> 1439103 T5578 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1439106 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439120 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439121 T5572 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1439121 T5572 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1439122 T5572 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1439127 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection live nodes:[] collections:{collection1=DocCollection(collection1)={
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "router":{"name":"compositeId"},
   [junit4]   2> 	  "maxShardsPerNode":"1",
   [junit4]   2> 	  "autoCreated":"true",
   [junit4]   2> 	  "shards":{
   [junit4]   2> 	    "shard1":{
   [junit4]   2> 	      "range":"80000000-ffffffff",
   [junit4]   2> 	      "state":"active",
   [junit4]   2> 	      "replicas":{}},
   [junit4]   2> 	    "shard2":{
   [junit4]   2> 	      "range":"0-7fffffff",
   [junit4]   2> 	      "state":"active",
   [junit4]   2> 	      "replicas":{}}}}}
   [junit4]   2> 1439131 T5571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1439131 T5580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1439157 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439261 T5571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1439262 T5580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1439276 T5578 oasc.ZkController.register We are http://127.0.0.1:50864/b_u/vk/collection1/ and leader is http://127.0.0.1:50864/b_u/vk/collection1/
   [junit4]   2> 1439277 T5578 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50864/b_u/vk
   [junit4]   2> 1439277 T5578 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1439277 T5578 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1439278 T5578 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1439280 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439281 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439281 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439292 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50864/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50864_b_u%2Fvk",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1439299 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439406 T5580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1439407 T5571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1440701 T5549 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1440702 T5549 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1440707 T5549 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50869
   [junit4]   2> 1440708 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1440709 T5549 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1440709 T5549 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003
   [junit4]   2> 1440709 T5549 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/'
   [junit4]   2> 1440750 T5549 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/solr.xml
   [junit4]   2> 1440766 T5549 oasc.CoreContainer.<init> New CoreContainer 97661228
   [junit4]   2> 1440767 T5549 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/]
   [junit4]   2> 1440767 T5549 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1440768 T5549 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1440768 T5549 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1440768 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1440769 T5549 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1440769 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1440770 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1440771 T5549 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1440771 T5549 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1440772 T5549 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1440772 T5549 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1440772 T5549 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1440773 T5549 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50861/solr
   [junit4]   2> 1440773 T5549 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1440775 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1440778 T5593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7684637b name:ZooKeeperConnection Watcher:127.0.0.1:50861 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1440779 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1440783 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1440786 T5595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20362662 name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1440786 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1440804 T5549 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1441821 T5549 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50869_b_u%2Fvk
   [junit4]   2> 1441827 T5549 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50869_b_u%2Fvk
   [junit4]   2> 1441876 T5596 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1441876 T5596 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1441882 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1441883 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1441884 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1441891 T5596 oasc.ZkController.preRegister Registering watch for collection collection1
   [junit4]   2> 1441894 T5596 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1441899 T5596 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1441905 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50869/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50869_b_u%2Fvk",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1441908 T5596 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1441923 T5572 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1441925 T5572 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1441939 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1441940 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1441941 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1441943 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 1 
   [junit4]   2> 1442911 T5596 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1442912 T5596 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1442917 T5596 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1442917 T5596 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1442923 T5596 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1442924 T5596 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/collection1/'
   [junit4]   2> 1442928 T5596 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1442929 T5596 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1442931 T5596 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1442996 T5596 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1443008 T5596 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1443011 T5596 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1443015 T5596 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1443136 T5596 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1443138 T5596 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1443139 T5596 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1443144 T5596 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1443200 T5596 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1443202 T5596 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1443202 T5596 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-003/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1/
   [junit4]   2> 1443203 T5596 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@600bc100
   [junit4]   2> 1443204 T5596 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1
   [junit4]   2> 1443205 T5596 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1/index/
   [junit4]   2> 1443206 T5596 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1443208 T5596 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1/index
   [junit4]   2> 1443209 T5596 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=83.3212890625, floorSegmentMB=1.5, forceMergeDeletesPctAllowed=3.1730657031306997, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1443214 T5596 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1443215 T5596 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1443222 T5596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1443223 T5596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1443224 T5596 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1443224 T5596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1443225 T5596 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1443225 T5596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1443226 T5596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1443226 T5596 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1443226 T5596 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1443228 T5596 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1443228 T5596 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1443229 T5596 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1443229 T5596 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1443230 T5596 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1443230 T5596 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1443232 T5596 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1443233 T5596 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1443242 T5596 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1443245 T5596 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1443245 T5596 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1443246 T5596 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5752913056219086]
   [junit4]   2> 1443248 T5596 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1443248 T5596 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1443249 T5596 oass.SolrIndexSearcher.<init> Opening Searcher@3586e30c[collection1] main
   [junit4]   2> 1443249 T5596 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1443252 T5596 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1443252 T5596 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1443253 T5596 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1443253 T5596 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1443254 T5596 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1443257 T5596 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1443257 T5596 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1443257 T5596 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1443257 T5596 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1443263 T5597 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3586e30c[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1443266 T5596 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1443268 T5600 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50869/b_u/vk collection:collection1 shard:shard2
   [junit4]   2> 1443269 T5549 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1443270 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1443270 T5600 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1443287 T5600 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1443291 T5600 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1443292 T5600 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1115 name=collection1 org.apache.solr.core.SolrCore@271a3632 url=http://127.0.0.1:50869/b_u/vk/collection1 node=127.0.0.1:50869_b_u%2Fvk C1115_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50869/b_u/vk, core=collection1, node_name=127.0.0.1:50869_b_u%2Fvk}
   [junit4]   2> 1443292 T5600 C1115 P50869 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50869/b_u/vk/collection1/
   [junit4]   2> 1443292 T5600 C1115 P50869 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1443293 T5600 C1115 P50869 oasc.SyncStrategy.syncToMe http://127.0.0.1:50869/b_u/vk/collection1/ has no replicas
   [junit4]   2> 1443293 T5600 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50869/b_u/vk/collection1/ shard2
   [junit4]   2> 1443293 T5600 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1443293 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443308 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443356 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1443360 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1443363 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 2 
   [junit4]   2> 1443387 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443388 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1443392 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1443394 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 3 
   [junit4]   2> 1443413 T5600 oasc.ZkController.register We are http://127.0.0.1:50869/b_u/vk/collection1/ and leader is http://127.0.0.1:50869/b_u/vk/collection1/
   [junit4]   2> 1443414 T5600 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50869/b_u/vk
   [junit4]   2> 1443414 T5600 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1443414 T5600 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1443415 T5600 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1443418 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443433 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50869/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50869_b_u%2Fvk",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1443447 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443449 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1443453 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1443457 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 4 
   [junit4]   2> 1444985 T5549 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1444987 T5549 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1444992 T5549 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50872
   [junit4]   2> 1444993 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1444994 T5549 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1444995 T5549 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004
   [junit4]   2> 1444995 T5549 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/'
   [junit4]   2> 1445040 T5549 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/solr.xml
   [junit4]   2> 1445055 T5549 oasc.CoreContainer.<init> New CoreContainer 1448378605
   [junit4]   2> 1445056 T5549 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/]
   [junit4]   2> 1445056 T5549 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1445057 T5549 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1445057 T5549 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1445057 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1445058 T5549 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1445058 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1445058 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1445059 T5549 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1445059 T5549 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1445060 T5549 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1445060 T5549 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1445060 T5549 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1445061 T5549 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50861/solr
   [junit4]   2> 1445061 T5549 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1445063 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1445068 T5611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42db43c8 name:ZooKeeperConnection Watcher:127.0.0.1:50861 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1445068 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1445074 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1445077 T5613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@342f7d4b name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1445077 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1445095 T5549 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1446113 T5549 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50872_b_u%2Fvk
   [junit4]   2> 1446116 T5549 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50872_b_u%2Fvk
   [junit4]   2> 1446136 T5614 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1446136 T5614 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1446138 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1446141 T5614 oasc.ZkController.preRegister Registering watch for collection collection1
   [junit4]   2> 1446141 T5614 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1446142 T5614 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1446145 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50872/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50872_b_u%2Fvk",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1446146 T5614 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1446155 T5572 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1446158 T5572 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1446173 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1446174 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1446175 T5613 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1446176 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1446178 T5613 oascc.ZkStateReader$6.process Updating data for collection1 to ver 5 
   [junit4]   2> 1446178 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 5 
   [junit4]   2> 1447148 T5614 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1447149 T5614 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1447150 T5614 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1447150 T5614 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1447153 T5614 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1447153 T5614 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/collection1/'
   [junit4]   2> 1447155 T5614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1447155 T5614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1447156 T5614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1447195 T5614 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1447206 T5614 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1447211 T5614 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1447216 T5614 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1447333 T5614 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1447334 T5614 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1447336 T5614 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1447340 T5614 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1447388 T5614 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1447389 T5614 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1447389 T5614 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-004/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2/
   [junit4]   2> 1447389 T5614 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@600bc100
   [junit4]   2> 1447391 T5614 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2
   [junit4]   2> 1447392 T5614 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2/index/
   [junit4]   2> 1447393 T5614 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1447394 T5614 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2/index
   [junit4]   2> 1447395 T5614 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=83.3212890625, floorSegmentMB=1.5, forceMergeDeletesPctAllowed=3.1730657031306997, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1447400 T5614 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1447400 T5614 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1447406 T5614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1447407 T5614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1447407 T5614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1447408 T5614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1447409 T5614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1447409 T5614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1447410 T5614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1447410 T5614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1447410 T5614 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1447411 T5614 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1447412 T5614 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1447412 T5614 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1447412 T5614 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1447413 T5614 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1447413 T5614 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1447414 T5614 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1447415 T5614 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1447424 T5614 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1447426 T5614 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1447426 T5614 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1447427 T5614 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5752913056219086]
   [junit4]   2> 1447429 T5614 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1447429 T5614 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1447429 T5614 oass.SolrIndexSearcher.<init> Opening Searcher@1019145c[collection1] main
   [junit4]   2> 1447430 T5614 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1447433 T5614 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1447433 T5614 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1447436 T5614 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1447436 T5614 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1447437 T5614 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1447440 T5614 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1447440 T5614 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1447440 T5614 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1447441 T5614 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1447445 T5615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1019145c[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1447446 T5614 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1447448 T5618 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50872/b_u/vk collection:collection1 shard:shard1
   [junit4]   2> 1447449 T5549 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1447449 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1447450 T5618 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1447460 T5618 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1447470 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1447471 T5618 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1447471 T5618 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1116 name=collection1 org.apache.solr.core.SolrCore@23fbc3e5 url=http://127.0.0.1:50872/b_u/vk/collection1 node=127.0.0.1:50872_b_u%2Fvk C1116_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50872/b_u/vk, core=collection1, node_name=127.0.0.1:50872_b_u%2Fvk}
   [junit4]   2> 1447471 T5618 C1116 P50872 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50872/b_u/vk/collection1/
   [junit4]   2> 1447472 T5618 C1116 P50872 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1447472 T5618 C1116 P50872 oasc.SyncStrategy.syncToMe http://127.0.0.1:50872/b_u/vk/collection1/ has no replicas
   [junit4]   2> 1447472 T5618 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50872/b_u/vk/collection1/ shard1
   [junit4]   2> 1447473 T5618 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1447487 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1447495 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1447497 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1447498 T5613 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1447499 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 6 
   [junit4]   2> 1447501 T5613 oascc.ZkStateReader$6.process Updating data for collection1 to ver 6 
   [junit4]   2> 1447586 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1447590 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1447601 T5613 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1447602 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1447606 T5613 oascc.ZkStateReader$6.process Updating data for collection1 to ver 7 
   [junit4]   2> 1447606 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 7 
   [junit4]   2> 1447647 T5618 oasc.ZkController.register We are http://127.0.0.1:50872/b_u/vk/collection1/ and leader is http://127.0.0.1:50872/b_u/vk/collection1/
   [junit4]   2> 1447648 T5618 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50872/b_u/vk
   [junit4]   2> 1447648 T5618 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1447648 T5618 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1447648 T5618 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1447653 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1447665 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50872/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50872_b_u%2Fvk",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1447685 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1447686 T5572 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1447688 T5595 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1447691 T5595 oascc.ZkStateReader$6.process Updating data for collection1 to ver 8 
   [junit4]   2> 1447691 T5613 oascc.ZkStateReader$6.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1447694 T5613 oascc.ZkStateReader$6.process Updating data for collection1 to ver 8 
   [junit4]   2> 1449187 T5549 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1449188 T5549 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1449195 T5549 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50875
   [junit4]   2> 1449196 T5549 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1449197 T5549 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1449197 T5549 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-005
   [junit4]   2> 1449197 T5549 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-005/'
   [junit4]   2> 1449235 T5549 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-005/solr.xml
   [junit4]   2> 1449250 T5549 oasc.CoreContainer.<init> New CoreContainer 301732944
   [junit4]   2> 1449251 T5549 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-005/]
   [junit4]   2> 1449251 T5549 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1449252 T5549 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1449252 T5549 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1449253 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1449253 T5549 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1449254 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1449254 T5549 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1449254 T5549 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1449254 T5549 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1449255 T5549 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1449257 T5549 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1449257 T5549 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1449257 T5549 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50861/solr
   [junit4]   2> 1449258 T5549 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1449259 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1449263 T5629 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ed72869 name:ZooKeeperConnection Watcher:127.0.0.1:50861 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1449263 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1449269 T5549 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1449271 T5631 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3913ee37 name:ZooKeeperConnection Watcher:127.0.0.1:50861/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1449272 T5549 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1449290 T5549 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1450308 T5549 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50875_b_u%2Fvk
   [junit4]   2> 1450314 T5549 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50875_b_u%2Fvk
   [junit4]   2> 1450360 T5632 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1450360 T5632 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1450365 T5571 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1450371 T5632 oasc.ZkController.preRegister Registering watch for collection collection1
   [junit4]   2> 1450371 T5632 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1450375 T5632 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1450382 T5632 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1450382 T5572 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50875/b_u/vk",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50875_b_u%2Fvk",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1450393 T5572 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1450396 T5

[...truncated too long message...]

checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1492273 T5714 C1125 P50897 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 1492274 T5714 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 1492274 T5714 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4175083d
   [junit4]   2> 1492286 T5714 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=55,cumulative_deletesById=19,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=6184,transaction_logs_total_number=1}
   [junit4]   2> 1492286 T5714 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1492286 T5714 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1492290 T5714 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1492294 T5714 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1492295 T5714 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1492296 T5714 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty7 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty7;done=false>>]
   [junit4]   2> 1492297 T5714 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty7
   [junit4]   2> 1492297 T5714 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty7/index [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty7/index;done=false>>]
   [junit4]   2> 1492297 T5714 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001/tempDir-001/jetty7/index
   [junit4]   2> 1492298 T5714 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1492298 T5714 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-D2E263878DC21E9D-001
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=et_EE, timezone=America/El_Salvador
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=2,threads=1,free=127694728,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [ReturnFieldsTest, TestCodecSupport, TestXIncludeConfig, SpellPossibilityIteratorTest, AnalysisErrorHandlingTest, TestOverriddenPrefixQueryForCustomFieldType, TestManagedSchemaFieldResource, DirectUpdateHandlerTest, PingRequestHandlerTest, TestIBSimilarityFactory, ZkCLITest, ResponseLogComponentTest, SampleTest, PrimUtilsTest, TestReloadAndDeleteDocs, TestShardHandlerFactory, SolrXmlInZkTest, RemoteQueryErrorTest, TestSolrDeletionPolicy2, TestStressLucene, TestSearcherReuse, PathHierarchyTokenizerFactoryTest, ResourceLoaderTest, SimplePostToolTest, SearchHandlerTest, BasicZkTest, TestRangeQuery, CopyFieldTest, FieldMutatingUpdateProcessorTest, AliasIntegrationTest, TestHashPartitioner, TestShortCircuitedRequests, TestReplicationHandlerBackup, SolrRequestParserTest, TestFaceting, TermsComponentTest, TestUtils, TestReRankQParserPlugin, AsyncMigrateRouteKeyTest, DateFieldTest, NotRequiredUniqueKeyTest, TestCopyFieldCollectionResource, RequiredFieldsTest, TestPhraseSuggestions, DistributedExpandComponentTest, AddBlockUpdateTest, AbstractAnalyticsFacetTest, ModifyConfFileTest, MBeansHandlerTest, NoCacheHeaderTest, SuggesterTest, TestIndexingPerformance, TestClassNameShortening, XmlUpdateRequestHandlerTest, MultiThreadedOCPTest, SolrTestCaseJ4Test, TestPostingsSolrHighlighter, IndexSchemaTest, SliceStateTest, EnumFieldTest, TestQuerySenderNoQuery, TestImplicitCoreProperties, BJQParserTest, FieldFacetTest, AbstractAnalyticsStatsTest, CachingDirectoryFactoryTest, CurrencyFieldOpenExchangeTest, CollectionsAPIDistributedZkTest, BasicDistributedZkTest, TestAtomicUpdateErrorCases, BadCopyFieldTest, StatsComponentTest, SimpleFacetsTest, TestRecovery, TestAddFieldRealTimeGet, SoftAutoCommitTest, CoreAdminCreateDiscoverTest, TestNRTOpen, TestDynamicFieldResource, CollectionsAPIAsyncDistributedZkTest, TestFreeTextSuggestions, BadIndexSchemaTest, TestCollationField, SpellingQueryConverterTest, ClusterStateUpdateTest, TestTrie, AnalyticsQueryTest, DocumentBuilderTest, TestManagedResource, RangeFacetTest, CoreAdminHandlerTest, HighlighterTest, ChaosMonkeySafeLeaderTest, URLClassifyProcessorTest, QueryParsingTest, CacheHeaderTest, HighlighterMaxOffsetTest, SolrInfoMBeanTest, QueryElevationComponentTest, SOLR749Test, TestFastLRUCache, TestAnalyzeInfixSuggestions, ExternalCollectionsTest, BinaryUpdateRequestHandlerTest, EchoParamsTest, UUIDFieldTest, DistribDocExpirationUpdateProcessorTest, TestRecoveryHdfs, TestMiniSolrCloudCluster, TestStressUserVersions, TestArbitraryIndexDir, DistributedSpellCheckComponentTest, TestPHPSerializedResponseWriter, TestHighlightDedupGrouping, UniqFieldsUpdateProcessorFactoryTest, TestMaxScoreQueryParser, TestDistributedMissingSort, HdfsUnloadDistributedZkTest, DistributedQueryComponentOptimizationTest, TestFastWriter, CSVRequestHandlerTest, TestOmitPositions, TestModifyConfFiles, NumericFieldsTest, ShardSplitTest, TestInitQParser, TestFastOutputStream, OverseerTest, TestSweetSpotSimilarityFactory, TestChildDocTransformer, StandardRequestHandlerTest, SynonymTokenizerTest, CircularListTest, TestFunctionQuery, TestSolrIndexConfig, ConvertedLegacyTest, TestSuggestSpellingConverter, QueryResultKeyTest, DeleteReplicaTest, ZkNodePropsTest, UnloadDistributedZkTest, HdfsDirectoryTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed in 57.39s, 1 test, 1 failure <<< FAILURES!

[...truncated 830 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1298: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:922: There were test failures: 416 suites, 1705 tests, 1 failure, 43 ignored (16 assumptions)

Total time: 105 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message