lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_45) - Build # 8814 - Failure!
Date Wed, 01 Jan 2014 14:01:27 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8814/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([88C1DA80D709C0D3:8CC95573C5AC2FF2]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10301 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 211718 T716 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1388583825561
   [junit4]   2> 211718 T716 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 211721 T716 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 211721 T716 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 211722 T717 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 211822 T716 oasc.ZkTestServer.run start zk server on port:55628
   [junit4]   2> 211823 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212102 T724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aed8d11 name:ZooKeeperConnection Watcher:127.0.0.1:55628/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212102 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212104 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212108 T727 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47b4d50c name:ZooKeeperConnection Watcher:127.0.0.1:55628 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212108 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212114 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212123 T730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73aefd4b name:ZooKeeperConnection Watcher:127.0.0.1:55628 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212124 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212124 T716 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 212131 T716 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 212138 T716 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 212143 T716 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 212147 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212153 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212156 T733 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f734b19 name:ZooKeeperConnection Watcher:127.0.0.1:55628/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212156 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212159 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 212164 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 212175 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 212180 T716 oasc.Overseer.start Overseer (id=91002229592358915-127.0.0.1:55628_solr-n_0000000000) starting
   [junit4]   2> 212185 T716 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 212193 T716 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 212200 T716 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 212206 T735 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 212208 T734 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 212211 T716 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 212213 T734 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 212214 T734 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 212214 T734 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 212219 T733 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 212221 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 212264 T716 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 212266 T733 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 212268 T734 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 212269 T734 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 212269 T734 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 212269 T734 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 212272 T733 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 212377 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 212423 T716 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55628 55628
   [junit4]   2> 212515 T716 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 212521 T716 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 212522 T716 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 212522 T736 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 212622 T716 oasc.ZkTestServer.run start zk server on port:59571
   [junit4]   2> 212623 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212626 T743 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e79def4 name:ZooKeeperConnection Watcher:127.0.0.1:59571 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212626 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212646 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212647 T746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51279627 name:ZooKeeperConnection Watcher:127.0.0.1:59571 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212648 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212648 T716 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 212653 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212654 T749 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fcfa632 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212654 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212655 T716 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 212657 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212659 T752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fa3d873 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212659 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212661 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 212664 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 212670 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 212673 T716 oasc.Overseer.start Overseer (id=91002229644918787-127.0.0.1:59571_solr-n_0000000000) starting
   [junit4]   2> 212675 T716 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 212679 T716 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 212684 T716 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 212688 T754 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 212688 T716 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 212689 T753 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 212693 T716 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 212697 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212704 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212707 T757 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@452cff4a name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212708 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212711 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212717 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 212722 T757 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 212722 T749 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 212725 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212727 T760 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16da6b8e name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212727 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212731 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212736 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 212742 T757 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 212742 T749 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 212742 T760 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 212744 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212747 T763 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59c53c0f name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212747 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212750 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212755 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 212760 T757 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 212760 T749 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 212761 T763 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 212761 T760 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 212763 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212765 T766 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aabeda9 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212766 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212769 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212775 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 212781 T760 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 212781 T763 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 212782 T749 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 212782 T757 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 212782 T766 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 212784 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212786 T769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74c37ce2 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212787 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212790 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212797 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 212802 T749 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 212803 T757 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 212803 T763 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 212803 T766 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 212803 T769 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 212803 T760 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 212805 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212807 T772 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b4c317 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212808 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212810 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212816 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 212821 T766 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212821 T757 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212821 T763 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212821 T769 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212821 T749 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212821 T772 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212821 T760 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 212822 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212824 T775 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56eba9d0 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212824 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212826 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212828 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 212832 T757 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212832 T769 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212832 T763 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212832 T749 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212833 T766 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212833 T760 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212832 T772 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212833 T775 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 212833 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212835 T778 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fef6218 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212836 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212837 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212839 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 212842 T769 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212842 T757 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212842 T749 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212843 T763 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212843 T760 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212843 T772 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212843 T778 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212843 T775 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212843 T766 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 212844 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212845 T781 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f91a4c9 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212845 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212847 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212850 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 212853 T766 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212853 T772 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212853 T778 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212853 T760 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212854 T763 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212853 T749 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212854 T775 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212854 T769 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212854 T757 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212854 T781 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 212855 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212856 T784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13d365fb name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212856 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212858 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212861 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 212864 T757 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212866 T763 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T778 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T781 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T775 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T772 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T769 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T760 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T784 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T749 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212867 T766 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 212868 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212870 T787 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d39bb6f name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212870 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212872 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212875 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 212880 T772 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212880 T781 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212880 T769 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212880 T760 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212881 T757 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212881 T787 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212880 T763 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212881 T784 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212881 T749 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212881 T766 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212880 T778 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212880 T775 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 212882 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212884 T790 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f56202 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212884 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212886 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212890 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 212894 T757 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212894 T781 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212894 T778 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212894 T784 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212895 T749 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212894 T775 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212895 T766 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212896 T760 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212896 T763 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212896 T772 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212896 T787 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212897 T790 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212896 T769 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 212897 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212899 T793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b137b09 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212900 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212902 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212906 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 212913 T749 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212913 T790 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212914 T778 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212914 T769 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212914 T781 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212914 T775 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212914 T793 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212914 T772 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212915 T766 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212915 T787 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212915 T763 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212915 T760 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212916 T784 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212916 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212916 T757 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 212918 T796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ad18cb name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212918 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212920 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212925 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 212932 T775 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212932 T760 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212932 T784 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212933 T763 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212933 T766 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212934 T790 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212934 T778 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212934 T769 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212934 T772 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212935 T781 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212935 T796 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212935 T787 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212936 T793 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212936 T749 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212935 T757 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 212937 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212939 T799 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@548555dc name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212939 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212943 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212948 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 212955 T766 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212956 T787 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212956 T790 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212956 T772 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212956 T778 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212957 T749 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212956 T793 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212957 T784 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212957 T781 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212957 T760 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212958 T763 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212957 T757 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212957 T775 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212959 T799 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212958 T796 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212958 T769 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 212960 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212963 T802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b4da670 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212963 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212966 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212971 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 212978 T790 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212979 T749 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212979 T766 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212980 T757 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212980 T793 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212980 T772 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212980 T781 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212980 T775 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212980 T799 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T784 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T769 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T796 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T802 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T763 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T778 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T760 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212981 T787 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 212982 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212984 T805 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36e88ef0 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212984 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212986 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 212988 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 212992 T749 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212993 T796 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212992 T763 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212993 T778 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212993 T784 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T772 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T769 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212993 T793 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T787 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T805 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T766 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212995 T775 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T799 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T781 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212995 T802 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T790 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212994 T757 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212996 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 212994 T760 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 212997 T808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdd6d1 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 212997 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 212999 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213002 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 213007 T796 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213007 T799 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213007 T802 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213007 T775 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213007 T760 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213008 T805 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213008 T784 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213008 T769 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213008 T790 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213008 T763 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213009 T781 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213009 T772 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213009 T778 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213010 T808 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213010 T757 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213010 T766 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213010 T749 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213010 T787 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213010 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213010 T793 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 213012 T811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6497e29a name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213012 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213013 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213016 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 213019 T766 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213019 T778 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213019 T772 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213019 T787 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213020 T781 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213019 T799 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213020 T790 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213020 T760 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213022 T802 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213022 T749 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213023 T769 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213023 T757 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213023 T775 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213023 T805 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213024 T796 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213023 T784 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213023 T793 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213024 T811 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213024 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213024 T763 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213024 T808 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 213026 T814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4efd6462 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213027 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213028 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213031 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 213034 T808 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213034 T781 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213034 T778 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213034 T749 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213034 T787 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213034 T799 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213034 T784 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213035 T772 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213035 T766 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T757 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T805 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213035 T796 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T763 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213037 T769 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T775 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213037 T811 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T802 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T760 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213035 T790 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213036 T793 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213039 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213040 T814 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 213041 T817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50df1046 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213041 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213043 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213045 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 213049 T775 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T802 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T763 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213049 T811 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T760 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T796 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T799 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T769 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T808 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T790 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T784 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T787 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213050 T805 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T772 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213052 T781 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213052 T749 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T766 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T793 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T817 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213051 T814 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213052 T778 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213052 T757 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 213052 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213054 T820 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bf813a name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213054 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213056 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213058 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 213062 T808 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213062 T793 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213062 T749 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213062 T778 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T766 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T769 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T820 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T802 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T811 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T757 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T805 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T781 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T775 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213064 T760 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213063 T790 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213062 T796 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213065 T817 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213065 T763 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213064 T772 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213064 T787 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213065 T814 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213065 T799 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213065 T784 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 213065 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213067 T823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1536e3d7 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213067 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213068 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213071 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 213075 T757 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T778 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T793 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T811 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T775 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T781 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T820 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T787 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T769 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T799 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T784 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213075 T772 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213075 T749 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213078 T802 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213077 T814 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213077 T790 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213077 T808 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213077 T763 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213077 T760 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T796 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T805 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213076 T766 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213080 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213081 T826 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52c59936 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213081 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213081 T823 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213082 T817 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 213082 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213084 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 213089 T805 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213089 T802 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T811 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T781 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T787 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T790 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T760 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T775 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T772 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T757 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T823 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T769 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213090 T799 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T784 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T826 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213092 T778 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213092 T808 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T796 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T749 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T763 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T814 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213091 T793 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213092 T820 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213092 T817 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213092 T766 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 213093 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213094 T829 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38c457c name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213094 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213096 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213098 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 213103 T781 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213103 T793 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213103 T787 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213103 T790 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213104 T763 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213104 T769 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213104 T799 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213104 T826 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213104 T772 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T757 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T808 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213104 T778 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T802 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T820 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T811 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T784 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T749 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T805 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T796 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T760 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213105 T775 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T829 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T817 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T814 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T766 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213106 T823 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 213107 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213108 T832 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aefd176 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213109 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213110 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213112 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 213117 T802 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T799 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T769 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T760 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T784 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T775 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T817 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213118 T811 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T749 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T778 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T793 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T814 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T808 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T790 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T820 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T766 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T763 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T805 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T781 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213119 T772 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213121 T823 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213121 T829 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213121 T826 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T757 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T832 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213120 T796 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213121 T787 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 213122 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213123 T835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f9118cd name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213123 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213125 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213129 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 213134 T820 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213134 T808 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213134 T766 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213134 T793 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T817 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T802 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T814 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T778 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T784 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T775 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T811 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T769 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T749 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T796 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213137 T832 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T772 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213137 T763 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213135 T790 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213137 T760 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213137 T826 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213137 T799 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T823 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213136 T757 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213138 T835 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213138 T829 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213138 T805 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213138 T787 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213137 T781 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 213139 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213140 T838 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39340fff name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213140 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213141 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213143 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 213148 T784 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213148 T802 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213148 T769 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213148 T817 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213148 T775 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213149 T811 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213149 T778 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213149 T760 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213149 T838 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213149 T793 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213149 T799 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T766 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T749 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T805 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T814 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T820 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T808 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T772 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213151 T796 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T790 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213150 T763 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213151 T829 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213151 T781 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213151 T832 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213152 T823 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213152 T757 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213157 T826 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213152 T787 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213157 T835 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 213158 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213159 T841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3de072a2 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213159 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213160 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213162 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 213168 T805 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213168 T796 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213168 T832 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213168 T763 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213168 T829 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T835 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T838 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T826 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T760 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213170 T775 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T781 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213170 T772 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213170 T784 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213170 T757 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213170 T799 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T769 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213173 T766 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213169 T811 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213173 T802 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213173 T790 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213171 T823 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213170 T787 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213174 T817 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213174 T808 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213174 T749 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213174 T820 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213174 T778 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213175 T814 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213175 T793 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213176 T841 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 213176 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213180 T844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52b6b9d6 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213180 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213182 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213185 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 213189 T799 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213190 T775 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213190 T760 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T749 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213190 T766 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T790 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T802 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T820 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T784 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T796 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T817 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T838 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T835 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T793 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T823 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213193 T805 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213193 T832 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T781 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213191 T778 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213193 T757 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213194 T844 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213194 T841 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213193 T811 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213193 T769 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213193 T763 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T826 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T829 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T787 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213192 T814 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213194 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213194 T772 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213194 T808 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 213199 T847 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4318b8f7 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213199 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213201 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213203 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 213207 T763 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213207 T781 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T787 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T826 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213209 T793 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213209 T799 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213209 T802 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T796 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T784 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T835 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T790 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T805 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T766 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213208 T844 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T817 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T811 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T823 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T772 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T829 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T778 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T820 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T814 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T775 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213210 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213210 T808 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213209 T769 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213209 T838 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213212 T850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b0d2cf5 name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213212 T847 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T841 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T749 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T832 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T760 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213211 T757 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 213213 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213215 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213218 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 213224 T769 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213224 T760 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213224 T784 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213224 T811 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213225 T802 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213225 T799 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213225 T817 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213225 T749 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213226 T814 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213226 T820 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213225 T775 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213226 T808 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213226 T790 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213226 T841 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T793 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T766 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T778 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T805 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T850 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T772 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T763 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213227 T796 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T829 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T832 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T757 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T835 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T844 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T838 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T823 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T826 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213229 T716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 213228 T787 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213228 T847 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213231 T781 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 213232 T853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@227f934b name:ZooKeeperConnection Watcher:127.0.0.1:59571/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 213232 T716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 213233 T716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 213235 T716 oascc.SolrZkClient.makePath makePath: /live_nodes/node32
   [junit4]   2> 213241 T790 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T814 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T749 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T793 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T778 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T850 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T760 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213243 T817 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T772 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213243 T769 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T838 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T841 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213242 T820 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T757 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213243 T775 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213243 T811 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T787 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T781 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213243 T784 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213245 T832 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213243 T766 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213245 T796 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213245 T829 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T805 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T823 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T844 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T835 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213244 T826 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213246 T853 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213245 T847 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 213245 T76

[...truncated too long message...]

IgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 232009 T1259 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 232011 T716 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 232011 T1268 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:654)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:649)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:649)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:649)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:875)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:226)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:187)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	
   [junit4]   2> 232015 T1267 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 232017 T1268 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 232018 T1259 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 232018 T1250 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 232024 T716 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51614 51614
   [junit4]   2> 232201 T716 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=88C1DA80D709C0D3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lt -Dtests.timezone=US/Central -Dtests.file.encoding=UTF-8
   [junit4] ERROR   3.11s J0 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([88C1DA80D709C0D3:8CC95573C5AC2FF2]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 232216 T716 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 232216 T716 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 233518 T1259 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 233519 T1259 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 235217 T716 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=lt, timezone=US/Central
   [junit4]   2> NOTE: Linux 3.8.0-34-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=1,free=87414392,total=167170048
   [junit4]   2> NOTE: All tests run in this JVM: [URLClassifyProcessorTest, LegacyHTMLStripCharFilterTest, TestSolrXmlPersistor, TestRandomDVFaceting, DocumentBuilderTest, SuggesterTest, SuggesterFSTTest, TestFunctionQuery, TestRandomMergePolicy, ShowFileRequestHandlerTest, TestValueSourceCache, TestLMDirichletSimilarityFactory, TestCoreContainer, SearchHandlerTest, TestSchemaSimilarityResource, DateMathParserTest, MBeansHandlerTest, FileBasedSpellCheckerTest, ScriptEngineTest, DeleteShardTest, QueryEqualityTest, DistributedDebugComponentTest, TestCollationFieldDocValues, CacheHeaderTest, InfoHandlerTest, ChangedSchemaMergeTest, PolyFieldTest, HdfsCollectionsAPIDistributedZkTest, FieldMutatingUpdateProcessorTest, SOLR749Test, DocValuesTest, TestRealTimeGet, BinaryUpdateRequestHandlerTest, TestPseudoReturnFields, OverseerCollectionProcessorTest, ResourceLoaderTest, SynonymTokenizerTest, QueryParsingTest, TestExtendedDismaxParser, SimpleFacetsTest, ClusterStateUpdateTest, TestCharFilters, OverseerTest]
   [junit4] Completed on J0 in 23.52s, 8 tests, 1 error <<< FAILURES!

[...truncated 802 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1307: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:940: There were test failures: 348 suites, 1522 tests, 1 error, 35 ignored (6 assumptions)

Total time: 44 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message