lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_55) - Build # 10248 - Failure!
Date Fri, 23 May 2014 21:30:22 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10248/
Java: 32bit/jdk1.7.0_55 -client -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard2 is not consistent.  Got 436 from https://127.0.0.1:59524/collection1lastClient and got 403 from https://127.0.0.1:60464/collection1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 436 from https://127.0.0.1:59524/collection1lastClient and got 403 from https://127.0.0.1:60464/collection1
	at __randomizedtesting.SeedInfo.seed([B91220D754653460:38F4AECF233A545C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11421 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/init-core-data-001
   [junit4]   2> 1158445 T3121 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1158445 T3121 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1158449 T3121 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1158449 T3121 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1158450 T3122 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1158550 T3121 oasc.ZkTestServer.run start zk server on port:41840
   [junit4]   2> 1158551 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1158587 T3128 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@312c7b name:ZooKeeperConnection Watcher:127.0.0.1:41840 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1158587 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1158588 T3121 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1158596 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1158598 T3130 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ee0ad name:ZooKeeperConnection Watcher:127.0.0.1:41840/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1158598 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1158599 T3121 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1158604 T3121 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1158609 T3121 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1158612 T3121 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1158617 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1158618 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1158625 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1158626 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1158635 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1158636 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1158640 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1158641 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1158646 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1158647 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1158651 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1158652 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1158655 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1158656 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1158661 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1158662 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1158665 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1158666 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1158669 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1158670 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1158674 T3121 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1158675 T3121 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1158681 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1158683 T3132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16f7a09 name:ZooKeeperConnection Watcher:127.0.0.1:41840/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1158683 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1158952 T3121 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1158963 T3121 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1158966 T3121 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60711
   [junit4]   2> 1158968 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1158968 T3121 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1158968 T3121 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002
   [junit4]   2> 1158969 T3121 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/'
   [junit4]   2> 1158982 T3121 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/solr.xml
   [junit4]   2> 1159026 T3121 oasc.CoreContainer.<init> New CoreContainer 18432408
   [junit4]   2> 1159027 T3121 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/]
   [junit4]   2> 1159028 T3121 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1159028 T3121 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1159028 T3121 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1159029 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1159029 T3121 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1159029 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1159030 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1159030 T3121 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1159030 T3121 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1159032 T3121 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1159033 T3121 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1159033 T3121 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1159033 T3121 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41840/solr
   [junit4]   2> 1159034 T3121 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1159035 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1159036 T3143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d18d5e name:ZooKeeperConnection Watcher:127.0.0.1:41840 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1159037 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1159039 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1159039 T3145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@198785f name:ZooKeeperConnection Watcher:127.0.0.1:41840/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1159040 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1159041 T3121 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1159044 T3121 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1159046 T3121 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1159048 T3121 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1159050 T3121 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1159052 T3121 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1159053 T3121 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60711_
   [junit4]   2> 1159055 T3121 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60711_
   [junit4]   2> 1159058 T3121 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1159059 T3121 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1159069 T3121 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60711_
   [junit4]   2> 1159070 T3121 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1159074 T3121 oasc.Overseer.start Overseer (id=91808051506380804-127.0.0.1:60711_-n_0000000000) starting
   [junit4]   2> 1159079 T3121 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1159090 T3147 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1159091 T3121 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1159093 T3121 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1159094 T3121 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1159098 T3146 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1159108 T3148 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1159108 T3148 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1159109 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1159109 T3148 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1159111 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1159111 T3146 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60711",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60711_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1159112 T3146 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1159112 T3146 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1159113 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1159114 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1160110 T3148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1160111 T3148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1160112 T3148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1160112 T3148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1160114 T3148 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1160114 T3148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/collection1/'
   [junit4]   2> 1160116 T3148 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1160116 T3148 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1160117 T3148 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1160178 T3148 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1160237 T3148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1160239 T3148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1160246 T3148 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1160666 T3148 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1160669 T3148 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1160671 T3148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1160675 T3148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1160702 T3148 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1160703 T3148 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1160712 T3148 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data/
   [junit4]   2> 1160712 T3148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc449c
   [junit4]   2> 1160713 T3148 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data
   [junit4]   2> 1160713 T3148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data/index/
   [junit4]   2> 1160714 T3148 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1160714 T3148 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data/index
   [junit4]   2> 1160715 T3148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3726176567927949]
   [junit4]   2> 1160720 T3148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1160721 T3148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1160727 T3148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1160727 T3148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1160728 T3148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1160729 T3148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1160729 T3148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1160730 T3148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1160731 T3148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1160731 T3148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1160732 T3148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1160733 T3148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1160734 T3148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1160734 T3148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1160735 T3148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1160736 T3148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1160737 T3148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1160738 T3148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1160743 T3148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1160748 T3148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1160749 T3148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1160751 T3148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=226155120, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1160753 T3148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1160753 T3148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1160754 T3148 oass.SolrIndexSearcher.<init> Opening Searcher@43bc35[collection1] main
   [junit4]   2> 1160754 T3148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1160756 T3148 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1160756 T3148 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1160758 T3148 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1160758 T3148 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1160758 T3148 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1160759 T3148 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1160759 T3148 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1160760 T3148 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1160760 T3148 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1160765 T3149 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43bc35[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1160767 T3148 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1160768 T3152 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60711 collection:control_collection shard:shard1
   [junit4]   2> 1160768 T3121 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1160769 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1160769 T3152 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1160772 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1160773 T3154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@680a21 name:ZooKeeperConnection Watcher:127.0.0.1:41840/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1160773 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1160776 T3152 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1160776 T3121 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1160779 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160780 T3152 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1160780 T3152 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1160780 T3121 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1160781 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C2824 name=collection1 org.apache.solr.core.SolrCore@18c8344 url=https://127.0.0.1:60711/collection1 node=127.0.0.1:60711_ C2824_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:60711, core=collection1, node_name=127.0.0.1:60711_}
   [junit4]   2> 1160780 T3152 C2824 P60711 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60711/collection1/
   [junit4]   2> 1160782 T3152 C2824 P60711 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1160782 T3152 C2824 P60711 oasc.SyncStrategy.syncToMe https://127.0.0.1:60711/collection1/ has no replicas
   [junit4]   2> 1160782 T3152 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60711/collection1/ shard1
   [junit4]   2> 1160782 T3152 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1160783 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160784 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1160784 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1160787 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160788 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1160790 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160892 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1160892 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1160938 T3152 oasc.ZkController.register We are https://127.0.0.1:60711/collection1/ and leader is https://127.0.0.1:60711/collection1/
   [junit4]   2> 1160939 T3152 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60711
   [junit4]   2> 1160939 T3152 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1160939 T3152 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1160939 T3152 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1160940 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160940 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160941 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160941 T3152 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1160942 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1160942 T3146 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60711",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60711_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1160944 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1161045 T3121 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1161046 T3121 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1161046 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1161046 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1161050 T3121 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1161051 T3121 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48775
   [junit4]   2> 1161054 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1161055 T3121 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1161056 T3121 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003
   [junit4]   2> 1161057 T3121 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/'
   [junit4]   2> 1161083 T3121 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/solr.xml
   [junit4]   2> 1161148 T3121 oasc.CoreContainer.<init> New CoreContainer 11699996
   [junit4]   2> 1161149 T3121 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/]
   [junit4]   2> 1161150 T3121 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1161150 T3121 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1161151 T3121 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1161151 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1161151 T3121 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1161152 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1161152 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1161152 T3121 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1161153 T3121 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1161155 T3121 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1161155 T3121 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1161155 T3121 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1161156 T3121 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41840/solr
   [junit4]   2> 1161156 T3121 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1161157 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1161160 T3165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@126b8f3 name:ZooKeeperConnection Watcher:127.0.0.1:41840 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1161161 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1161165 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1161167 T3167 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20812f name:ZooKeeperConnection Watcher:127.0.0.1:41840/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1161167 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1161176 T3121 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1162178 T3121 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48775_
   [junit4]   2> 1162180 T3121 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48775_
   [junit4]   2> 1162185 T3154 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1162185 T3167 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1162186 T3145 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1162197 T3168 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1162198 T3168 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1162204 T3168 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1162204 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1162205 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1162205 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1162207 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1162209 T3146 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:48775",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48775_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1162209 T3146 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1162210 T3146 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1162213 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1162316 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1162316 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1162316 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1163205 T3168 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1163205 T3168 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1163206 T3168 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1163207 T3168 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1163209 T3168 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1163210 T3168 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/collection1/'
   [junit4]   2> 1163211 T3168 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1163212 T3168 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1163213 T3168 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1163275 T3168 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1163323 T3168 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1163325 T3168 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1163333 T3168 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1163781 T3168 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1163784 T3168 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1163786 T3168 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1163794 T3168 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1163817 T3168 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1163817 T3168 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1163818 T3168 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1/
   [junit4]   2> 1163818 T3168 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc449c
   [junit4]   2> 1163819 T3168 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1
   [junit4]   2> 1163820 T3168 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1/index/
   [junit4]   2> 1163820 T3168 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1163822 T3168 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1/index
   [junit4]   2> 1163823 T3168 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3726176567927949]
   [junit4]   2> 1163827 T3168 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1163827 T3168 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1163830 T3168 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1163830 T3168 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1163831 T3168 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1163831 T3168 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1163831 T3168 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1163831 T3168 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1163832 T3168 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1163832 T3168 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1163832 T3168 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1163833 T3168 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1163833 T3168 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1163834 T3168 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1163834 T3168 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1163835 T3168 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1163835 T3168 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1163835 T3168 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1163838 T3168 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1163843 T3168 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1163843 T3168 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1163845 T3168 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=226155120, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1163847 T3168 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1163847 T3168 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1163848 T3168 oass.SolrIndexSearcher.<init> Opening Searcher@1159ff2[collection1] main
   [junit4]   2> 1163848 T3168 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1163849 T3168 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1163849 T3168 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1163850 T3168 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1163850 T3168 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1163850 T3168 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1163851 T3168 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1163852 T3168 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1163852 T3168 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1163852 T3168 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1163862 T3169 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1159ff2[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1163867 T3168 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1163869 T3172 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:48775 collection:collection1 shard:shard3
   [junit4]   2> 1163870 T3121 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1163871 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1163871 T3172 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 1163881 T3172 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 1163883 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1163884 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1163884 T3172 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1163885 T3172 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1163884 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C2825 name=collection1 org.apache.solr.core.SolrCore@9a18cb url=https://127.0.0.1:48775/collection1 node=127.0.0.1:48775_ C2825_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:48775, core=collection1, node_name=127.0.0.1:48775_}
   [junit4]   2> 1163885 T3172 C2825 P48775 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:48775/collection1/
   [junit4]   2> 1163886 T3172 C2825 P48775 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1163887 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1163887 T3172 C2825 P48775 oasc.SyncStrategy.syncToMe https://127.0.0.1:48775/collection1/ has no replicas
   [junit4]   2> 1163888 T3172 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:48775/collection1/ shard3
   [junit4]   2> 1163888 T3172 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 1163891 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1163894 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1163895 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1163894 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1163901 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1163905 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164007 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164007 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164007 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164052 T3172 oasc.ZkController.register We are https://127.0.0.1:48775/collection1/ and leader is https://127.0.0.1:48775/collection1/
   [junit4]   2> 1164052 T3172 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:48775
   [junit4]   2> 1164052 T3172 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1164052 T3172 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1164053 T3172 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1164054 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164054 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164054 T3172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1164054 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164056 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1164056 T3146 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:48775",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48775_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1164058 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164162 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164162 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164162 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164179 T3121 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1164180 T3121 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1164183 T3121 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1164185 T3121 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50683
   [junit4]   2> 1164188 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1164189 T3121 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1164189 T3121 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004
   [junit4]   2> 1164190 T3121 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/'
   [junit4]   2> 1164212 T3121 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/solr.xml
   [junit4]   2> 1164255 T3121 oasc.CoreContainer.<init> New CoreContainer 2216945
   [junit4]   2> 1164255 T3121 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/]
   [junit4]   2> 1164256 T3121 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1164257 T3121 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1164257 T3121 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1164257 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1164258 T3121 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1164258 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1164258 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1164258 T3121 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1164259 T3121 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1164261 T3121 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1164261 T3121 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1164261 T3121 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1164262 T3121 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41840/solr
   [junit4]   2> 1164262 T3121 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1164263 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1164265 T3183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bc0664 name:ZooKeeperConnection Watcher:127.0.0.1:41840 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1164265 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1164268 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1164270 T3185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f76bb0 name:ZooKeeperConnection Watcher:127.0.0.1:41840/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1164270 T3121 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1164279 T3121 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1165281 T3121 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50683_
   [junit4]   2> 1165283 T3121 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50683_
   [junit4]   2> 1165288 T3154 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1165288 T3185 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1165288 T3167 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1165288 T3145 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1165301 T3186 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1165301 T3186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1165303 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1165303 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1165303 T3186 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1165304 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1165306 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1165307 T3146 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50683",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50683_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1165307 T3146 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1165308 T3146 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1165311 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1165415 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1165415 T3185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1165415 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1165415 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1166305 T3186 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1166305 T3186 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1166307 T3186 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1166307 T3186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1166309 T3186 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1166309 T3186 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/collection1/'
   [junit4]   2> 1166311 T3186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1166311 T3186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1166312 T3186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1166381 T3186 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1166454 T3186 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1166456 T3186 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1166464 T3186 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1166902 T3186 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1166905 T3186 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1166906 T3186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1166911 T3186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1166942 T3186 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1166943 T3186 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1166944 T3186 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2/
   [junit4]   2> 1166944 T3186 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc449c
   [junit4]   2> 1166945 T3186 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2
   [junit4]   2> 1166946 T3186 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2/index/
   [junit4]   2> 1166947 T3186 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1166948 T3186 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2/index
   [junit4]   2> 1166949 T3186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3726176567927949]
   [junit4]   2> 1166957 T3186 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1166957 T3186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1166964 T3186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1166965 T3186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1166965 T3186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1166966 T3186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1166966 T3186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1166967 T3186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1166968 T3186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1166969 T3186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1166969 T3186 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1166970 T3186 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1166972 T3186 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1166972 T3186 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1166973 T3186 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1166974 T3186 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1166975 T3186 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1166977 T3186 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1166982 T3186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1166987 T3186 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1166988 T3186 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1166989 T3186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=226155120, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1166990 T3186 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1166990 T3186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1166990 T3186 oass.SolrIndexSearcher.<init> Opening Searcher@1ecccd8[collection1] main
   [junit4]   2> 1166991 T3186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1166992 T3186 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1166992 T3186 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1166993 T3186 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1166993 T3186 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1166993 T3186 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1166994 T3186 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1166994 T3186 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1166995 T3186 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1166995 T3186 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1167001 T3187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ecccd8[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1167006 T3186 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1167007 T3190 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50683 collection:collection1 shard:shard2
   [junit4]   2> 1167008 T3121 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1167008 T3190 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1167009 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1167019 T3190 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1167022 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167022 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167023 T3190 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1167024 T3190 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1167023 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167024 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C2826 name=collection1 org.apache.solr.core.SolrCore@15812fa url=https://127.0.0.1:50683/collection1 node=127.0.0.1:50683_ C2826_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50683, core=collection1, node_name=127.0.0.1:50683_}
   [junit4]   2> 1167024 T3190 C2826 P50683 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50683/collection1/
   [junit4]   2> 1167026 T3190 C2826 P50683 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1167026 T3190 C2826 P50683 oasc.SyncStrategy.syncToMe https://127.0.0.1:50683/collection1/ has no replicas
   [junit4]   2> 1167027 T3190 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50683/collection1/ shard2
   [junit4]   2> 1167027 T3190 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1167028 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167029 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167029 T3185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167029 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167029 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167031 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167032 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1167034 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167136 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167136 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167136 T3185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167136 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167182 T3190 oasc.ZkController.register We are https://127.0.0.1:50683/collection1/ and leader is https://127.0.0.1:50683/collection1/
   [junit4]   2> 1167182 T3190 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50683
   [junit4]   2> 1167183 T3190 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1167183 T3190 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1167183 T3190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1167184 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167184 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167185 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167184 T3190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1167186 T3146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1167187 T3146 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50683",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50683_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1167188 T3145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1167277 T3121 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1167278 T3121 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1167283 T3121 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1167285 T3121 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48491
   [junit4]   2> 1167287 T3121 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1167288 T3121 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1167288 T3121 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-005
   [junit4]   2> 1167289 T3121 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-005/'
   [junit4]   2> 1167291 T3145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167291 T3167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167291 T3185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167291 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1167306 T3121 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-005/solr.xml
   [junit4]   2> 1167376 T3121 oasc.CoreContainer.<init> New CoreContainer 8549253
   [junit4]   2> 1167376 T3121 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-005/]
   [junit4]   2> 1167377 T3121 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1167378 T3121 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1167378 T3121 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1167378 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1167379 T3121 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1167379 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1167379 T3121 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1167380 T3121 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1167380 T3121 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1167382 T3121 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1167382 T3121 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1167383 T3121 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1167383 T3121 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41840/solr
   [junit4]   2> 1167383 T3121 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1167384 T3121 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1167386 T3201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8c9f7c name:ZooKeeperConnection Watcher:127.0.0.1:41840 got event WatchedEvent state:SyncConnected ty

[...truncated too long message...]

9
   [junit4]   2> 1360232 T3934 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty5
   [junit4]   2> 1360239 T4081 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360237 T4202 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360236 T4058 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty10/index.20140524001700319 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty10/index.20140524001700319;done=false>>]
   [junit4]   2> 1360241 T4058 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-B91220D754653460-001/tempDir-001/jetty10/index.20140524001700319
   [junit4]   2> 1360241 T4202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1360240 T4081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
   [junit4]   2> 1360240 T3934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 9)
   [junit4]   2> 1360243 T4081 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360242 T4202 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360242 T4058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1360243 T3934 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360244 T4058 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360245 T3934 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360245 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360246 T3934 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1360246 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360247 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360248 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360248 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360248 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360249 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360249 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360249 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360250 T4058 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1360250 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360250 T4058 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=ar_SD, timezone=Europe/Kaliningrad
   [junit4]   2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.7.0_55 (32-bit)/cpus=8,threads=1,free=87638504,total=323600384
   [junit4]   2> NOTE: All tests run in this JVM: [TestDynamicFieldCollectionResource, SolrIndexConfigTest, TestComponentsName, TestFieldTypeResource, TestCSVLoader, TestNonDefinedSimilarityFactory, NoCacheHeaderTest, TestWordDelimiterFilterFactory, DistributedDebugComponentTest, TestRandomFaceting, DeleteShardTest, TestStressRecovery, MultiThreadedOCPTest, TestMultiCoreConfBootstrap, TestManagedResource, SpellCheckCollatorTest, StressHdfsTest, SpellingQueryConverterTest, SimplePostToolTest, OpenCloseCoreStressTest, CoreContainerCoreInitFailuresTest, SuggesterTSTTest, DebugComponentTest, TestRandomMergePolicy, TestAnalyzeInfixSuggestions, CachingDirectoryFactoryTest, TestSweetSpotSimilarityFactory, AsyncMigrateRouteKeyTest, TestManagedSynonymFilterFactory, DeleteReplicaTest, TestRangeQuery, OutputWriterTest, TestSolrQueryParserResource, TestMiniSolrCloudCluster, TestSolrXmlPersistence, TestManagedSchemaFieldResource, CursorPagingTest, TestStressLucene, SuggesterWFSTTest, PathHierarchyTokenizerFactoryTest, TestCollationFieldDocValues, FileUtilsTest, BasicZkTest, TestInitQParser, TestQuerySenderListener, ZkSolrClientTest, TestRecoveryHdfs, TestSerializedLuceneMatchVersion, UniqFieldsUpdateProcessorFactoryTest, SortByFunctionTest, TestCollationKeyRangeQueries, ShardSplitTest, ZkNodePropsTest, NumericFieldsTest, TestUpdate, TestFieldTypeCollectionResource, TestDefaultSimilarityFactory, TestCollectionAPI, HighlighterConfigTest, SolrPluginUtilsTest, TestQuerySenderNoQuery, FullSolrCloudDistribCmdsTest, HdfsCollectionsAPIDistributedZkTest, TestInfoStreamLogging, LeaderElectionIntegrationTest, TestRTGBase, DistanceFunctionTest, TestBinaryField, JSONWriterTest, TestSolrXml, TestDynamicFieldResource, TestFiltering, TestMaxScoreQueryParser, BasicDistributedZk2Test, TestSolr4Spatial, TermVectorComponentDistributedTest, TestSchemaVersionResource, TestHashPartitioner, DistributedExpandComponentTest, StandardRequestHandlerTest, TestSearcherReuse, TestJmxIntegration, TestIBSimilarityFactory, HdfsChaosMonkeySafeLeaderTest, TestStandardQParsers, TestZkChroot, TestCSVResponseWriter, TestSimpleQParserPlugin, TestSchemaResource, CurrencyFieldXmlFileTest, CursorMarkTest, TestFastOutputStream, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 202.08s, 1 test, 1 failure <<< FAILURES!

[...truncated 610 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 failure, 36 ignored (15 assumptions)

Total time: 72 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_55 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message