lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9646 - Failure!
Date Sun, 02 Mar 2014 08:53:28 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9646/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([3C1C75B87FE62E1F:3814FA4B6D43C13E]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:129)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:120)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:589)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:431)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:428)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:385)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:372)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:126)
	... 47 more




Build Log:
[...truncated 10718 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 787000 T3142 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 787001 T3142 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1393749753024
   [junit4]   2> 787001 T3142 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 787005 T3142 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 787005 T3142 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 787005 T3143 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 787105 T3142 oasc.ZkTestServer.run start zk server on port:58061
   [junit4]   2> 787106 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787163 T3149 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a3ab957 name:ZooKeeperConnection Watcher:127.0.0.1:58061/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787163 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787164 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787167 T3151 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f72cff8 name:ZooKeeperConnection Watcher:127.0.0.1:58061 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787167 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787169 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787170 T3153 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34e36903 name:ZooKeeperConnection Watcher:127.0.0.1:58061 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787170 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787170 T3142 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 787173 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 787175 T3142 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 787176 T3142 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 787177 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787180 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787181 T3155 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@201ff6cc name:ZooKeeperConnection Watcher:127.0.0.1:58061/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787181 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787182 T3142 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 787183 T3142 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 787186 T3142 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58061_solr
   [junit4]   2> 787186 T3142 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 787188 T3142 oasc.Overseer.start Overseer (id=91340783814574083-127.0.0.1:58061_solr-n_0000000000) starting
   [junit4]   2> 787189 T3142 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 787193 T3142 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 787196 T3142 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 787198 T3157 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 787199 T3157 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 787199 T3156 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 787200 T3142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 787201 T3156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 787201 T3156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 787201 T3156 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 787203 T3155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 787204 T3149 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 787251 T3142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 787253 T3155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 787254 T3156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 787255 T3156 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 787255 T3156 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 787255 T3156 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 787256 T3155 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 787361 T3149 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 787406 T3142 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58061 58061
   [junit4]   2> 787486 T3142 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 787493 T3142 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 787494 T3142 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 787494 T3158 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 787594 T3142 oasc.ZkTestServer.run start zk server on port:36804
   [junit4]   2> 787595 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787598 T3164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28695b60 name:ZooKeeperConnection Watcher:127.0.0.1:36804 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787599 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787611 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787613 T3166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60eedd49 name:ZooKeeperConnection Watcher:127.0.0.1:36804 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787613 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787613 T3142 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 787617 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787618 T3168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fabf9fc name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787618 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787619 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 787620 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787622 T3170 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78731b4d name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787622 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787622 T3142 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 787624 T3142 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 787627 T3142 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36804_solr
   [junit4]   2> 787627 T3142 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 787629 T3142 oasc.Overseer.start Overseer (id=91340783846817795-127.0.0.1:36804_solr-n_0000000000) starting
   [junit4]   2> 787630 T3142 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 787633 T3142 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 787637 T3142 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 787638 T3172 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 787639 T3142 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 787639 T3172 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 787640 T3171 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 787642 T3142 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 787643 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787648 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787650 T3174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130a8f67 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787650 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787651 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787653 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 787655 T3168 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 787655 T3174 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 787655 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787657 T3176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@741c3c07 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787659 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787660 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787662 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 787664 T3168 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 787664 T3174 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 787664 T3176 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 787665 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787666 T3178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10b020e1 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787666 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787667 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787679 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 787681 T3174 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 787682 T3168 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 787682 T3176 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 787682 T3178 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 787682 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787684 T3180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@791681d5 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787684 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787685 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787687 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 787689 T3174 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 787689 T3176 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 787689 T3168 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 787689 T3178 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 787689 T3180 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 787689 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787690 T3182 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e6fa9e7 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787691 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787692 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787693 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 787695 T3178 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 787695 T3176 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 787696 T3174 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 787696 T3182 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 787695 T3168 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 787695 T3180 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 787696 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787697 T3184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29ed2040 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787697 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787698 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787700 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 787701 T3178 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787702 T3168 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787702 T3182 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787702 T3174 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787702 T3184 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787702 T3176 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787702 T3180 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 787703 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787704 T3186 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7408fe11 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787704 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787705 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787706 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 787708 T3178 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3180 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3176 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3174 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3168 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787709 T3184 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3186 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787709 T3182 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 787710 T3188 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d60f50b name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787710 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787712 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787713 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 787717 T3176 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787717 T3178 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3188 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3174 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3186 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3168 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787718 T3180 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3182 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787718 T3184 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 787719 T3190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7818779d name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787720 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787721 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787722 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 787724 T3168 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3174 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3186 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3176 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3178 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3184 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787726 T3190 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787726 T3182 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3188 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787725 T3180 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 787726 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787727 T3192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f9f0a18 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787727 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787728 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787730 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 787733 T3180 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787733 T3192 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787733 T3178 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3188 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3176 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3184 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3168 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3182 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3174 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787734 T3186 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787735 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787734 T3190 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 787736 T3194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49452c41 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787736 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787737 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787739 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 787741 T3174 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787741 T3178 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3190 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3180 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3194 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787743 T3184 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787743 T3186 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3176 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3168 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3188 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3192 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787742 T3182 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 787743 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787744 T3196 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de125ff name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787745 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787746 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787747 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 787750 T3184 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787750 T3178 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787750 T3196 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787751 T3188 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787750 T3174 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787751 T3180 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787751 T3186 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787751 T3176 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787751 T3182 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787750 T3168 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787750 T3192 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787752 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787751 T3194 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787751 T3190 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 787753 T3198 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5af84083 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787753 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787754 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787756 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 787759 T3180 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787759 T3194 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787759 T3182 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787759 T3188 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787760 T3186 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787760 T3178 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3192 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787761 T3176 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3184 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3190 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3168 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3198 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3196 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787761 T3174 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 787763 T3200 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d7b389e name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787763 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787764 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787766 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 787770 T3184 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787770 T3168 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787770 T3196 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787770 T3198 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3176 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3180 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3188 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3182 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3192 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3174 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3178 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787772 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787772 T3190 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3194 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3186 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787771 T3200 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 787773 T3202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c8f2f69 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787774 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787775 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787777 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 787780 T3176 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787780 T3182 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787780 T3188 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787780 T3186 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3196 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787780 T3194 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3184 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3180 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3168 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787782 T3190 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3198 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3178 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787782 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787781 T3202 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3174 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787782 T3200 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787781 T3192 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 787784 T3204 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79cd0c88 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787784 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787785 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787787 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 787791 T3204 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3196 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3182 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3168 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3178 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3198 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3192 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3174 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3202 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3176 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3190 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787791 T3180 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787793 T3200 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787793 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787792 T3194 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3184 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3188 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787792 T3186 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 787795 T3206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49a2c962 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787795 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787796 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787797 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 787800 T3182 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787800 T3180 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3202 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3200 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3176 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787800 T3186 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3168 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3188 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3204 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3174 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787801 T3190 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3184 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3196 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3194 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787801 T3198 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3206 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3192 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787802 T3178 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 787805 T3208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ff73dc9 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787805 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787805 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787807 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 787809 T3200 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787809 T3194 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787809 T3202 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787809 T3184 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787809 T3192 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787809 T3186 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787810 T3174 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787809 T3168 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3204 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787810 T3196 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787810 T3190 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3176 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3206 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3178 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787812 T3208 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3180 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3182 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3198 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787811 T3188 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 787812 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787813 T3210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b42a42 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787814 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787814 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787816 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 787818 T3198 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787818 T3184 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787818 T3194 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3182 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3176 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3186 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3192 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3190 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3174 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3188 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3178 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3196 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3206 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787819 T3208 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787821 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787821 T3204 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3210 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3202 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3200 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3168 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787820 T3180 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 787822 T3212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@729dfdfb name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787822 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787823 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787825 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 787827 T3208 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787827 T3184 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787831 T3190 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787831 T3212 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787827 T3198 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787831 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787831 T3178 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787831 T3196 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787831 T3192 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3204 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3186 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787832 T3214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30ba1e3f name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787830 T3168 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3188 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3202 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3182 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3194 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3180 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3206 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3176 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3210 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3200 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787830 T3174 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 787832 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787835 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787836 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 787841 T3200 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787841 T3196 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3182 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3174 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3184 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3168 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3188 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3192 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3176 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3210 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3212 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3178 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3206 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3208 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3202 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3198 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3186 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787842 T3180 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787844 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787844 T3214 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787844 T3204 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787844 T3194 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787843 T3190 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 787845 T3216 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c71da4d name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787846 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787846 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787848 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 787851 T3206 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3190 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3210 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3202 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3188 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3196 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3184 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3178 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3214 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3182 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3176 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787853 T3216 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3168 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3208 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3174 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3200 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3198 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3192 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3186 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787852 T3180 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3194 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787851 T3204 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787853 T3212 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 787853 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787855 T3218 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58e70514 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787855 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787856 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787857 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 787860 T3190 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3194 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787860 T3206 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3178 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3204 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3184 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3198 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787860 T3180 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3208 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3210 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787860 T3176 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3214 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3212 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3168 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3196 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3182 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787862 T3200 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3188 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3192 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3218 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3216 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3174 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3202 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787861 T3186 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 787864 T3220 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75c2c7f6 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787863 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787865 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787866 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787867 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 787870 T3190 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3212 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3198 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3188 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3192 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3182 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3206 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3176 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3216 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3214 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3218 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3220 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3204 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3202 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3186 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3194 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3210 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3180 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3174 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787873 T3222 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ae247a name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787871 T3184 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787871 T3200 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3168 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3196 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3178 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787870 T3208 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 787872 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787874 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787875 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787877 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 787880 T3194 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3176 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3208 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3214 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3186 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3190 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3216 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3184 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3222 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787882 T3212 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3218 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787882 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787882 T3220 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3168 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3192 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3198 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3210 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3188 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3182 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3202 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3204 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787881 T3180 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3174 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3206 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3200 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3196 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787880 T3178 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 787883 T3224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b5595e9 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787885 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787886 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787888 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 787892 T3180 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3186 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3182 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3210 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3190 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3192 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3214 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3220 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3184 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787895 T3194 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3212 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3216 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3222 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3176 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787895 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787893 T3198 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3168 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3204 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3188 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3202 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787893 T3206 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787897 T3226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1777c41a name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787895 T3224 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787895 T3196 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787895 T3208 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3174 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3200 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3178 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787894 T3218 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 787897 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787899 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787901 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 787906 T3196 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787906 T3194 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787906 T3224 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787906 T3174 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3218 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787906 T3216 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3178 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3214 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3202 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3208 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787906 T3176 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3222 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3192 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3180 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3204 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3188 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3186 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3210 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3182 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3206 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3200 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787907 T3190 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787909 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787908 T3184 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3168 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3198 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3220 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3226 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787908 T3212 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 787911 T3228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68c7dd11 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787911 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787912 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787914 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 787917 T3214 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787917 T3178 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787917 T3200 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787917 T3174 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3196 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3208 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3176 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3216 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3194 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3218 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3224 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787918 T3228 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3168 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3220 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3198 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3212 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3188 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3184 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787922 T3180 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787922 T3222 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3192 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787922 T3206 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787922 T3204 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787922 T3190 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3186 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3202 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3210 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787921 T3226 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787923 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787924 T3230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a925bfc name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787924 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787925 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787927 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 787931 T3206 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787931 T3186 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787931 T3184 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787931 T3190 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787931 T3180 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3174 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3216 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787931 T3222 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787931 T3204 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3214 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3200 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3196 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3230 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787934 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787933 T3228 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3168 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3176 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787933 T3208 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3220 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3224 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3182 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 787932 T3218 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3178 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3194 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3198 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3212 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3226 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787937 T3232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6dc56ee4 name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787932 T3192 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3202 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3210 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787932 T3188 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787937 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787937 T3182 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 787939 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787940 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 787946 T3210 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3222 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3202 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3184 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3220 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3190 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3186 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3204 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3206 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3230 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3192 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3212 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3198 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3168 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787947 T3180 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3188 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787946 T3182 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787949 T3226 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787950 T3216 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787950 T3214 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787950 T3196 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787950 T3208 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787950 T3194 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787951 T3218 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787951 T3200 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787951 T3228 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787951 T3178 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787952 T3174 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787952 T3176 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787952 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787952 T3224 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787952 T3232 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 787954 T3234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cc22c9f name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787954 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787955 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787970 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 787978 T3200 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787978 T3214 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787978 T3196 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787978 T3178 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3174 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3224 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3228 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3208 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3218 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3176 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3194 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787980 T3220 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787980 T3184 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787980 T3232 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787980 T3168 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787981 T3212 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787979 T3216 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787981 T3198 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787981 T3230 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787981 T3210 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787982 T3192 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787982 T3226 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787982 T3182 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787983 T3202 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787983 T3204 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787983 T3190 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787983 T3206 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787984 T3186 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787984 T3222 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787984 T3180 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787984 T3188 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787984 T3234 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 787985 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 787988 T3236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f938fca name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 787989 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 787990 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 787992 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 787999 T3198 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3230 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3168 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3212 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3210 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3202 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3184 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3222 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3204 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3182 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3206 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3214 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3226 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3186 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3200 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3220 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788000 T3192 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3176 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3228 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3196 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3234 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788003 T3194 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3180 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788003 T3142 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 788004 T3224 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3188 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3236 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788001 T3190 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788003 T3232 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788003 T3178 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788005 T3238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e755ffc name:ZooKeeperConnection Watcher:127.0.0.1:36804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 788003 T3216 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3208 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3174 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788002 T3218 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 788005 T3142 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 788007 T3142 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 788009 T3142 oascc.SolrZkClient.makePath makePath: /live_nodes/node32
   [junit4]   2> 788014 T3224 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788014 T3216 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788014 T3218 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788014 T3194 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788014 T3176 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788015 T3214 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788015 T3232 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788015 T3238 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788015 T3196 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 788016 T3228 oascc.ZkStateReader$3.pr

[...truncated too long message...]

Client$2.execute(SolrZkClient.java:176)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:173)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:635)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:630)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:630)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:630)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:949)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:235)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:192)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	
   [junit4]   2> 808877 T3657 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 808878 T3644 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 808878 T3657 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 808881 T3142 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56049 56049
   [junit4]   2> 809186 T3142 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=3C1C75B87FE62E1F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_US -Dtests.timezone=Asia/Rangoon -Dtests.file.encoding=UTF-8
   [junit4] ERROR   3.24s J0 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3C1C75B87FE62E1F:3814FA4B6D43C13E]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:129)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:120)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:589)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:431)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:428)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:385)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:372)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:126)
   [junit4]    > 	... 47 more
   [junit4]   2> 809196 T3142 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 809196 T3142 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 810378 T3650 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 810379 T3650 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 812197 T3142 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=en_US, timezone=Asia/Rangoon
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=144417584,total=269959168
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrXMLSerializer, ExternalFileFieldSortTest, TestSchemaNameResource, DistributedQueryElevationComponentTest, HdfsDirectoryTest, CacheHeaderTest, TestCopyFieldCollectionResource, SolrIndexSplitterTest, TestLFUCache, FieldMutatingUpdateProcessorTest, TestPseudoReturnFields, BadComponentTest, TestInitQParser, TestAnalyzeInfixSuggestions, DistributedDebugComponentTest, TestLRUCache, NoFacetTest, TriLevelCompositeIdRoutingTest, BlockCacheTest, FieldFacetExtrasTest, TestSort, TestFaceting, SpellPossibilityIteratorTest, TestDistributedMissingSort, FileBasedSpellCheckerTest, ParsingFieldUpdateProcessorsTest, CursorPagingTest, PolyFieldTest, InfoHandlerTest, TestConfig, DistributedSpellCheckComponentTest, QueryResultKeyTest, SliceStateTest, TestPostingsSolrHighlighter, TestReversedWildcardFilterFactory, TestLuceneMatchVersion, BJQParserTest, ResponseLogComponentTest, SimpleFacetsTest, TestCloudManagedSchema, TestRemoteStreaming, TestSchemaVersionResource, TestSystemIdResolver, ChangedSchemaMergeTest, TestJmxMonitoredMap, StatelessScriptUpdateProcessorFactoryTest, CSVRequestHandlerTest, QueryElevationComponentTest, TestLMJelinekMercerSimilarityFactory, TestInfoStreamLogging, RegexBoostProcessorTest, TestPerFieldSimilarity, ShardRoutingTest, SynonymTokenizerTest, QueryParsingTest, TestElisionMultitermQuery, TestRandomMergePolicy, TestSolrQueryParserResource, LoggingHandlerTest, TestLMDirichletSimilarityFactory, CoreAdminHandlerTest, FastVectorHighlighterTest, SpatialFilterTest, ShowFileRequestHandlerTest, ResourceLoaderTest, ExpressionTest, TestQueryUtils, DateMathParserTest, SearchHandlerTest, AutoCommitTest, TestAddFieldRealTimeGet, UUIDFieldTest, AbstractAnalyticsStatsTest, TestPHPSerializedResponseWriter, OverseerCollectionProcessorTest, EchoParamsTest, TestSuggestSpellingConverter, RangeFacetTest, DOMUtilTest, TestDocBasedVersionConstraints, SOLR749Test, RecoveryZkTest, TestHighFrequencyDictionaryFactory, RemoteQueryErrorTest, TestBM25SimilarityFactory, MoreLikeThisHandlerTest, TestCoreContainer, QueryFacetTest, HdfsRecoveryZkTest, AbstractAnalyticsFacetTest, AnalysisErrorHandlingTest, BasicDistributedZkTest, BasicDistributedZk2Test, UnloadDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest]
   [junit4] Completed on J0 in 25.22s, 8 tests, 1 error <<< FAILURES!

[...truncated 573 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1611 tests, 1 error, 48 ignored (20 assumptions)

Total time: 51 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message