lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60) - Build # 10437 - Still Failing!
Date Sat, 14 Jun 2014 06:33:58 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10437/
Java: 64bit/jdk1.7.0_60 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 266 from http://127.0.0.1:55464/collection1lastClient and got 233 from http://127.0.0.1:52118/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 266 from http://127.0.0.1:55464/collection1lastClient and got 233 from http://127.0.0.1:52118/collection1
	at __randomizedtesting.SeedInfo.seed([BC5ED3981D1EAED:8A236321F68E8AD1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11176 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/init-core-data-001
   [junit4]   2> 610979 T1440 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 610980 T1440 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 610984 T1440 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 610984 T1440 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 610985 T1441 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 611084 T1440 oasc.ZkTestServer.run start zk server on port:57232
   [junit4]   2> 611085 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 611657 T1447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4710bfc4 name:ZooKeeperConnection Watcher:127.0.0.1:57232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 611657 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 611657 T1440 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 611661 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 611662 T1449 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17e63276 name:ZooKeeperConnection Watcher:127.0.0.1:57232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 611662 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 611662 T1440 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 611664 T1440 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 611666 T1440 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 611668 T1440 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 611669 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 611669 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 611672 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 611673 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 611675 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 611675 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 611678 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 611679 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 611681 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 611681 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 611683 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 611683 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 611685 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 611685 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 611687 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 611687 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 611689 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 611690 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 611691 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 611692 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 611694 T1440 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 611694 T1440 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 611944 T1440 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 611946 T1440 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57586
   [junit4]   2> 611947 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 611947 T1440 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 611947 T1440 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002
   [junit4]   2> 611948 T1440 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/'
   [junit4]   2> 611958 T1440 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/solr.xml
   [junit4]   2> 611979 T1440 oasc.CoreContainer.<init> New CoreContainer 1584429050
   [junit4]   2> 611979 T1440 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/]
   [junit4]   2> 611980 T1440 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 611980 T1440 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 611980 T1440 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 611980 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 611981 T1440 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 611981 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 611981 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 611981 T1440 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 611982 T1440 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 611982 T1440 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 611982 T1440 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 611982 T1440 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 611983 T1440 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57232/solr
   [junit4]   2> 611983 T1440 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 611984 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 611985 T1460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29d4ee86 name:ZooKeeperConnection Watcher:127.0.0.1:57232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 611986 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 611988 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 611989 T1462 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27b76648 name:ZooKeeperConnection Watcher:127.0.0.1:57232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 611990 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 611991 T1440 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 611994 T1440 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 611997 T1440 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 612000 T1440 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 612002 T1440 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 612004 T1440 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 612005 T1440 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57586_
   [junit4]   2> 612006 T1440 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57586_
   [junit4]   2> 612007 T1440 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 612009 T1440 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 612011 T1440 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57586_
   [junit4]   2> 612011 T1440 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 612013 T1440 oasc.Overseer.start Overseer (id=91929052833185795-127.0.0.1:57586_-n_0000000000) starting
   [junit4]   2> 612020 T1440 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 612027 T1464 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 612027 T1440 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 612029 T1440 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 612030 T1440 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 612032 T1463 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 612034 T1465 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 612034 T1465 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 612035 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 612035 T1465 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 612037 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 612037 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57586",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57586_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 612037 T1463 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 612038 T1463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 612040 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 612040 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 613036 T1465 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 613036 T1465 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 613037 T1465 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 613037 T1465 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 613038 T1465 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 613038 T1465 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/collection1/'
   [junit4]   2> 613039 T1465 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 613039 T1465 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 613039 T1465 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 613065 T1465 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 613086 T1465 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 613087 T1465 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 613094 T1465 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 613289 T1465 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 613291 T1465 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 613292 T1465 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 613295 T1465 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 613310 T1465 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 613310 T1465 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 613317 T1465 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data/
   [junit4]   2> 613317 T1465 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62eabb86
   [junit4]   2> 613317 T1465 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data
   [junit4]   2> 613318 T1465 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data/index/
   [junit4]   2> 613318 T1465 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 613318 T1465 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data/index
   [junit4]   2> 613319 T1465 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=94.2421875, floorSegmentMB=2.0185546875, forceMergeDeletesPctAllowed=7.432889642334867, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4705409414985543
   [junit4]   2> 613322 T1465 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 613322 T1465 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 613324 T1465 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 613325 T1465 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 613325 T1465 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 613325 T1465 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 613325 T1465 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 613325 T1465 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 613326 T1465 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 613326 T1465 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 613326 T1465 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 613326 T1465 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 613327 T1465 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 613327 T1465 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 613327 T1465 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 613327 T1465 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 613328 T1465 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 613328 T1465 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 613330 T1465 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 613331 T1465 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 613332 T1465 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 613332 T1465 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.818909405854101]
   [junit4]   2> 613333 T1465 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 613333 T1465 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 613334 T1465 oass.SolrIndexSearcher.<init> Opening Searcher@108613c[collection1] main
   [junit4]   2> 613334 T1465 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 613335 T1465 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 613335 T1465 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 613336 T1465 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 613336 T1465 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 613336 T1465 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 613337 T1465 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 613337 T1465 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 613337 T1465 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 613337 T1465 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 613339 T1466 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@108613c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 613340 T1465 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 613340 T1469 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57586 collection:control_collection shard:shard1
   [junit4]   2> 613341 T1440 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 613341 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 613341 T1469 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 613342 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 613343 T1471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@602baba9 name:ZooKeeperConnection Watcher:127.0.0.1:57232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 613343 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 613344 T1440 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 613346 T1469 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 613346 T1440 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 613347 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613348 T1469 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 613348 T1469 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1381 name=collection1 org.apache.solr.core.SolrCore@3c912727 url=http://127.0.0.1:57586/collection1 node=127.0.0.1:57586_ C1381_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:57586, core=collection1, node_name=127.0.0.1:57586_}
   [junit4]   2> 613348 T1469 C1381 P57586 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57586/collection1/
   [junit4]   2> 613348 T1469 C1381 P57586 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 613348 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 613349 T1469 C1381 P57586 oasc.SyncStrategy.syncToMe http://127.0.0.1:57586/collection1/ has no replicas
   [junit4]   2> 613349 T1469 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57586/collection1/ shard1
   [junit4]   2> 613349 T1469 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 613351 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613353 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613356 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613457 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 613457 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 613504 T1469 oasc.ZkController.register We are http://127.0.0.1:57586/collection1/ and leader is http://127.0.0.1:57586/collection1/
   [junit4]   2> 613505 T1469 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57586
   [junit4]   2> 613505 T1469 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 613505 T1469 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 613505 T1469 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 613507 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613507 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613507 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613507 T1469 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 613508 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 613509 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57586",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57586_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 613511 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 613580 T1440 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 613580 T1440 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 613582 T1440 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39025
   [junit4]   2> 613583 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 613583 T1440 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 613584 T1440 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003
   [junit4]   2> 613584 T1440 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/'
   [junit4]   2> 613594 T1440 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/solr.xml
   [junit4]   2> 613613 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 613613 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 613614 T1440 oasc.CoreContainer.<init> New CoreContainer 609134060
   [junit4]   2> 613615 T1440 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/]
   [junit4]   2> 613616 T1440 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 613616 T1440 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 613616 T1440 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 613616 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 613616 T1440 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 613617 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 613617 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 613617 T1440 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 613617 T1440 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 613618 T1440 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 613618 T1440 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 613618 T1440 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 613618 T1440 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57232/solr
   [junit4]   2> 613619 T1440 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 613619 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 613621 T1482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77b0441e name:ZooKeeperConnection Watcher:127.0.0.1:57232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 613621 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 613623 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 613624 T1484 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@462c3237 name:ZooKeeperConnection Watcher:127.0.0.1:57232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 613624 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 613630 T1440 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 614633 T1440 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39025_
   [junit4]   2> 614634 T1440 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39025_
   [junit4]   2> 614636 T1471 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 614636 T1462 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 614637 T1484 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 614642 T1485 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 614642 T1485 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 614643 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 614643 T1485 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 614644 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 614644 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 614645 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 614645 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39025",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39025_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 614645 T1463 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 614645 T1463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 614650 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 614752 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 614752 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 614753 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 615644 T1485 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 615644 T1485 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 615645 T1485 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 615646 T1485 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 615647 T1485 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 615648 T1485 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/collection1/'
   [junit4]   2> 615648 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 615649 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 615649 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 615693 T1485 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 615726 T1485 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 615728 T1485 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 615739 T1485 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 616012 T1485 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 616015 T1485 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 616016 T1485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 616020 T1485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 616039 T1485 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 616039 T1485 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 616039 T1485 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1/
   [junit4]   2> 616040 T1485 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62eabb86
   [junit4]   2> 616040 T1485 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1
   [junit4]   2> 616041 T1485 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1/index/
   [junit4]   2> 616041 T1485 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 616042 T1485 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1/index
   [junit4]   2> 616042 T1485 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=94.2421875, floorSegmentMB=2.0185546875, forceMergeDeletesPctAllowed=7.432889642334867, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4705409414985543
   [junit4]   2> 616047 T1485 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 616047 T1485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 616051 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 616051 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 616051 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 616051 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 616052 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 616052 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 616052 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 616052 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 616052 T1485 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 616053 T1485 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 616053 T1485 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 616053 T1485 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 616054 T1485 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 616054 T1485 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 616054 T1485 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 616055 T1485 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 616057 T1485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 616058 T1485 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 616058 T1485 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 616059 T1485 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.818909405854101]
   [junit4]   2> 616060 T1485 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 616060 T1485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 616061 T1485 oass.SolrIndexSearcher.<init> Opening Searcher@140dbbdb[collection1] main
   [junit4]   2> 616061 T1485 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 616062 T1485 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 616062 T1485 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 616062 T1485 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 616062 T1485 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 616063 T1485 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 616063 T1485 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 616063 T1485 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 616063 T1485 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 616064 T1485 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 616074 T1486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@140dbbdb[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 616076 T1485 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 616077 T1489 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39025 collection:collection1 shard:shard3
   [junit4]   2> 616077 T1440 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 616077 T1489 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 616077 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 616082 T1489 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 616083 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616084 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616084 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616084 T1489 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 616084 T1489 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1382 name=collection1 org.apache.solr.core.SolrCore@1eeb8f6e url=http://127.0.0.1:39025/collection1 node=127.0.0.1:39025_ C1382_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:39025, core=collection1, node_name=127.0.0.1:39025_}
   [junit4]   2> 616085 T1489 C1382 P39025 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39025/collection1/
   [junit4]   2> 616085 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 616085 T1489 C1382 P39025 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 616088 T1489 C1382 P39025 oasc.SyncStrategy.syncToMe http://127.0.0.1:39025/collection1/ has no replicas
   [junit4]   2> 616088 T1489 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39025/collection1/ shard3
   [junit4]   2> 616088 T1489 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 616089 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616093 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616098 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616202 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 616202 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 616202 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 616245 T1489 oasc.ZkController.register We are http://127.0.0.1:39025/collection1/ and leader is http://127.0.0.1:39025/collection1/
   [junit4]   2> 616245 T1489 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39025
   [junit4]   2> 616245 T1489 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 616246 T1489 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 616246 T1489 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 616247 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616247 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616247 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616247 T1489 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 616248 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 616249 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39025",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39025_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 616251 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 616313 T1440 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 616314 T1440 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 616316 T1440 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59640
   [junit4]   2> 616317 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 616317 T1440 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 616318 T1440 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004
   [junit4]   2> 616318 T1440 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/'
   [junit4]   2> 616328 T1440 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/solr.xml
   [junit4]   2> 616349 T1440 oasc.CoreContainer.<init> New CoreContainer 28888290
   [junit4]   2> 616349 T1440 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/]
   [junit4]   2> 616350 T1440 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 616350 T1440 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 616350 T1440 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 616351 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 616351 T1440 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 616351 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 616351 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 616351 T1440 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 616352 T1440 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 616352 T1440 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 616352 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 616353 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 616352 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 616352 T1440 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 616353 T1440 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 616354 T1440 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57232/solr
   [junit4]   2> 616354 T1440 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 616355 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 616356 T1500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@628a101f name:ZooKeeperConnection Watcher:127.0.0.1:57232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 616357 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 616359 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 616360 T1502 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13d172cd name:ZooKeeperConnection Watcher:127.0.0.1:57232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 616360 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 616367 T1440 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 617370 T1440 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59640_
   [junit4]   2> 617372 T1440 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59640_
   [junit4]   2> 617376 T1471 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 617377 T1502 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 617377 T1484 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 617377 T1462 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 617388 T1503 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 617389 T1503 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 617390 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 617391 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 617391 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 617390 T1503 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 617393 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 617395 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59640",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59640_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 617395 T1463 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 617395 T1463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 617403 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 617506 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 617506 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 617506 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 617506 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618392 T1503 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 618392 T1503 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 618393 T1503 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 618393 T1503 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 618394 T1503 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 618394 T1503 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/collection1/'
   [junit4]   2> 618395 T1503 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 618395 T1503 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 618395 T1503 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 618423 T1503 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 618443 T1503 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 618445 T1503 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 618467 T1503 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 618657 T1503 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 618659 T1503 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 618660 T1503 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 618663 T1503 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 618682 T1503 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 618683 T1503 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 618683 T1503 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2/
   [junit4]   2> 618683 T1503 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62eabb86
   [junit4]   2> 618684 T1503 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2
   [junit4]   2> 618684 T1503 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2/index/
   [junit4]   2> 618684 T1503 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 618685 T1503 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2/index
   [junit4]   2> 618685 T1503 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=94.2421875, floorSegmentMB=2.0185546875, forceMergeDeletesPctAllowed=7.432889642334867, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4705409414985543
   [junit4]   2> 618689 T1503 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 618699 T1503 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 618702 T1503 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 618702 T1503 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 618703 T1503 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 618703 T1503 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 618703 T1503 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 618703 T1503 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 618703 T1503 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 618704 T1503 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 618704 T1503 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 618704 T1503 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 618705 T1503 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 618705 T1503 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 618705 T1503 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 618705 T1503 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 618706 T1503 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 618706 T1503 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 618708 T1503 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 618709 T1503 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 618710 T1503 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 618710 T1503 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.818909405854101]
   [junit4]   2> 618711 T1503 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 618711 T1503 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 618712 T1503 oass.SolrIndexSearcher.<init> Opening Searcher@23ca8fc8[collection1] main
   [junit4]   2> 618712 T1503 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 618713 T1503 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 618713 T1503 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 618714 T1503 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 618714 T1503 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 618714 T1503 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 618715 T1503 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 618715 T1503 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 618715 T1503 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 618715 T1503 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 618717 T1504 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23ca8fc8[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 618720 T1503 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 618721 T1507 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59640 collection:collection1 shard:shard1
   [junit4]   2> 618722 T1440 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 618722 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 618722 T1507 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 618726 T1507 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 618728 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618728 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618728 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618728 T1507 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 618728 T1507 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1383 name=collection1 org.apache.solr.core.SolrCore@61331e0f url=http://127.0.0.1:59640/collection1 node=127.0.0.1:59640_ C1383_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:59640, core=collection1, node_name=127.0.0.1:59640_}
   [junit4]   2> 618729 T1507 C1383 P59640 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59640/collection1/
   [junit4]   2> 618729 T1507 C1383 P59640 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 618729 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 618729 T1507 C1383 P59640 oasc.SyncStrategy.syncToMe http://127.0.0.1:59640/collection1/ has no replicas
   [junit4]   2> 618729 T1507 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59640/collection1/ shard1
   [junit4]   2> 618729 T1507 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 618731 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618733 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618736 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618842 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618842 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618842 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618842 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618884 T1507 oasc.ZkController.register We are http://127.0.0.1:59640/collection1/ and leader is http://127.0.0.1:59640/collection1/
   [junit4]   2> 618884 T1507 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59640
   [junit4]   2> 618884 T1507 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 618885 T1507 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 618885 T1507 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 618886 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618886 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618887 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618887 T1507 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 618888 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 618889 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59640",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59640_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 618891 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 618946 T1440 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 618947 T1440 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 618950 T1440 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58778
   [junit4]   2> 618950 T1440 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 618950 T1440 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 618950 T1440 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-005
   [junit4]   2> 618951 T1440 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-005/'
   [junit4]   2> 618961 T1440 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-005/solr.xml
   [junit4]   2> 618981 T1440 oasc.CoreContainer.<init> New CoreContainer 1942939489
   [junit4]   2> 618982 T1440 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-005/]
   [junit4]   2> 618982 T1440 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 618982 T1440 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 618983 T1440 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 618983 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 618983 T1440 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 618983 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 618983 T1440 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 618984 T1440 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 618984 T1440 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 618984 T1440 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 618984 T1440 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 618985 T1440 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 618985 T1440 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57232/solr
   [junit4]   2> 618985 T1440 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 618986 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 618988 T1518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14b8ccad name:ZooKeeperConnection Watcher:127.0.0.1:57232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 618989 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 618992 T1440 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 618993 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618993 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618993 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618993 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 618993 T1520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4310a589 name:ZooKeeperConnection Watcher:127.0.0.1:57232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 618994 T1440 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 619000 T1440 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 620002 T1440 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58778_
   [junit4]   2> 620013 T1440 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58778_
   [junit4]   2> 620023 T1502 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 620023 T1462 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 620023 T1520 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 620023 T1484 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 620023 T1471 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 620029 T1521 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 620030 T1521 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 620031 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620031 T1521 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 620031 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620031 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620032 T1463 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 620033 T1463 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58778",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58778_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 620033 T1463 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 620033 T1463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 620037 T1462 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 620139 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 620139 T1462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 620139 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 620139 T1471 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 620139 T1520 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 621031 T1521 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 621031 T1521 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 621032 T1521 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 621032 T1521 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 621033 T1521 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 621033 T1521 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-005/collection1/'
   [junit4]   2> 621034 T1521 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-005/collection1/lib/.svn/' to classloader
   [junit4]   2> 621034 T1521 oasc.

[...truncated too long message...]

nExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 722216 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722218 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722218 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722219 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722219 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722216 T2091 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 722219 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722220 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722220 T2042 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722220 T1987 C2415 P37816 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9/index.20140614060815419,segFN=segments_7,generation=7}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9/index.20140614060815419,segFN=segments_8,generation=8}
   [junit4]   2> 722221 T1987 C2415 P37816 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 722242 T1987 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 722243 T1987 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 722243 T1987 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9;done=false>>]
   [junit4]   2> 722244 T1987 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9
   [junit4]   2> 722244 T1987 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9/index.20140614060815419 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9/index.20140614060815419;done=false>>]
   [junit4]   2> 722244 T1987 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001/tempDir-001/jetty9/index.20140614060815419
   [junit4]   2> 722245 T1987 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 722245 T1987 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-BC5ED3981D1EAED-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene49), sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ko, timezone=Iceland
   [junit4]   2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=8,threads=1,free=117950760,total=463876096
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressUserVersions, TestFieldSortValues, TestHighFrequencyDictionaryFactory, HighlighterMaxOffsetTest, SolrCoreCheckLockOnStartupTest, TestMultiCoreConfBootstrap, TestFoldingMultitermQuery, IndexSchemaRuntimeFieldTest, DisMaxRequestHandlerTest, HdfsDirectoryTest, DebugComponentTest, TestDefaultSearchFieldResource, OpenCloseCoreStressTest, DistributedQueryComponentOptimizationTest, SolrCmdDistributorTest, TestFreeTextSuggestions, DocExpirationUpdateProcessorFactoryTest, TestQueryTypes, TestFileDictionaryLookup, DirectSolrConnectionTest, TestDistributedMissingSort, TestOverriddenPrefixQueryForCustomFieldType, TestRequestStatusCollectionAPI, DocumentAnalysisRequestHandlerTest, TestSimpleQParserPlugin, TestSolrXmlPersistor, FieldAnalysisRequestHandlerTest, MigrateRouteKeyTest, SignatureUpdateProcessorFactoryTest, IndexBasedSpellCheckerTest, TestWordDelimiterFilterFactory, TestBinaryResponseWriter, SortByFunctionTest, TestRTGBase, BlockDirectoryTest, ZkSolrClientTest, TestCopyFieldCollectionResource, PreAnalyzedFieldTest, TestUtils, TestJmxIntegration, BasicDistributedZk2Test, TestFieldResource, ShardRoutingCustomTest, TestAnalyzedSuggestions, HdfsBasicDistributedZk2Test, RemoteQueryErrorTest, TestAtomicUpdateErrorCases, HdfsBasicDistributedZkTest, TestStressReorder, ParsingFieldUpdateProcessorsTest, TestIndexingPerformance, PluginInfoTest, InfoHandlerTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 112.23s, 1 test, 1 failure <<< FAILURES!

[...truncated 889 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 405 suites, 1654 tests, 1 failure, 38 ignored (17 assumptions)

Total time: 72 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message