lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.10-Linux (32bit/jdk1.8.0_20) - Build # 196 - Failure!
Date Mon, 29 Dec 2014 10:11:29 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.10-Linux/196/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC (asserts: false)

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
	at __randomizedtesting.SeedInfo.seed([57697C37D97D73C:17E18306F32381D]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11900 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-core/test/J1/./solr.cloud.OverseerTest-57697C37D97D73C-001/init-core-data-001
   [junit4]   2> 1391801 T4289 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1391801 T4289 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1391801 T4289 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1391804 T4289 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 1391805 T4289 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1391806 T4290 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1391906 T4289 oasc.ZkTestServer.run start zk server on port:48508
   [junit4]   2> 1391907 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1391909 T4296 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13610ec name:ZooKeeperConnection Watcher:127.0.0.1:48508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1391909 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1391910 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1391915 T4298 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6026f1 name:ZooKeeperConnection Watcher:127.0.0.1:48508 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1391916 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1391918 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1391919 T4300 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1086729 name:ZooKeeperConnection Watcher:127.0.0.1:48508 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1391919 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1391920 T4289 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1391923 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1391925 T4289 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1391926 T4289 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1391927 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1391929 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1391930 T4302 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127321e name:ZooKeeperConnection Watcher:127.0.0.1:48508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1391931 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1391932 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1391933 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1391941 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1391943 T4306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@97f126 name:ZooKeeperConnection Watcher:127.0.0.1:48508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1391944 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1391944 T4289 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
   [junit4]   2> 1391945 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1391947 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1391948 T4289 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1391949 T4289 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48508_solr
   [junit4]   2> 1391950 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1391951 T4289 oasc.Overseer.start Overseer (id=93051049187868676-127.0.0.1:48508_solr-n_0000000000) starting
   [junit4]   2> 1391952 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1391954 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1391955 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1391956 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1391958 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1391960 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1391963 T4289 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1391963 T4308 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1391964 T4307 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1391968 T4307 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1391968 T4307 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 1391969 T4307 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1391977 T4304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1391977 T4310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1391978 T4303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1392465 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1392470 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1392472 T4310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1393968 T4308 oasc.OverseerCollectionProcessor.amILeader  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:514)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:272)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1393968 T4308 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=93051049187868676-127.0.0.1:48508_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 1393973 T4307 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:446)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:243)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:237)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1393974 T4307 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48508_solr
   [junit4]   2> 1393974 T4304 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1393975 T4311 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:90)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
   [junit4]   2> 
   [junit4]   2> 1393976 T4289 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48508 48508
   [junit4]   2> 1394520 T4290 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48508 48508
   [junit4]   2> 1394522 T4289 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   2> 1394522 T4289 oasc.Overseer.close Overseer (id=93051049187868676-127.0.0.1:48508_solr-n_0000000000) closing
   [junit4]   2> 1394526 T4289 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 1394526 T4289 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1394527 T4312 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1394627 T4289 oasc.ZkTestServer.run start zk server on port:48061
   [junit4]   2> 1394628 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1394630 T4318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1590aeb name:ZooKeeperConnection Watcher:127.0.0.1:48061/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1394630 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1394631 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1394632 T4320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ac0546 name:ZooKeeperConnection Watcher:127.0.0.1:48061 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1394632 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1394633 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1394635 T4322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13779af name:ZooKeeperConnection Watcher:127.0.0.1:48061 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1394636 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1394636 T4289 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1394639 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1394642 T4289 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1394643 T4289 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1394644 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1394646 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1394647 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1394649 T4324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@82aaee name:ZooKeeperConnection Watcher:127.0.0.1:48061/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1394649 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1394649 T4289 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
   [junit4]   2> 1394650 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1394651 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1394652 T4289 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1394654 T4289 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48061_solr
   [junit4]   2> 1394654 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1394655 T4289 oasc.Overseer.start Overseer (id=93051049366257667-127.0.0.1:48061_solr-n_0000000000) starting
   [junit4]   2> 1394656 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1394658 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1394660 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1394661 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1394664 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1394667 T4289 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1394667 T4326 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1394669 T4325 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 1394669 T4325 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1394670 T4328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1394671 T4325 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1394672 T4328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1394672 T4325 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1394674 T4325 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1394675 T4329 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1394675 T4328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1394770 T4289 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48061 48061
   [junit4]   2> 1395312 T4312 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48061 48061
   [junit4]   2> 1395313 T4289 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 1395313 T4289 oasc.Overseer.close Overseer (id=93051049366257667-127.0.0.1:48061_solr-n_0000000000) closing
   [junit4]   2> 1395314 T4325 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48061_solr
   [junit4]   2> 1395316 T4330 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:90)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
   [junit4]   2> 
   [junit4]   2> 1395319 T4289 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 1395320 T4289 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1395320 T4331 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1395420 T4289 oasc.ZkTestServer.run start zk server on port:46568
   [junit4]   2> 1395421 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1395424 T4337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c9ef87 name:ZooKeeperConnection Watcher:127.0.0.1:46568/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1395424 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1395426 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1395427 T4339 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@664d8d name:ZooKeeperConnection Watcher:127.0.0.1:46568 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1395427 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1395430 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1395431 T4341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42a26f name:ZooKeeperConnection Watcher:127.0.0.1:46568 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1395431 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1395431 T4289 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1395433 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1395435 T4289 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1395437 T4289 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1395438 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1395440 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1395441 T4343 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14ca2d3 name:ZooKeeperConnection Watcher:127.0.0.1:46568/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1395441 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1395442 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1395444 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1395446 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1395447 T4347 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13dd46a name:ZooKeeperConnection Watcher:127.0.0.1:46568/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1395447 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1395448 T4289 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
   [junit4]   2> 1395449 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1395451 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1395452 T4289 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1395453 T4289 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46568_solr
   [junit4]   2> 1395454 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1395455 T4289 oasc.Overseer.start Overseer (id=93051049418162180-127.0.0.1:46568_solr-n_0000000000) starting
   [junit4]   2> 1395456 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1395459 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1395461 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1395462 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1395463 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1395465 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1395467 T4289 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1395468 T4349 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1395468 T4348 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1395470 T4348 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1395470 T4348 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1395470 T4348 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1395471 T4351 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1395471 T4345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1395471 T4344 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1395969 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1395976 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1395980 T4351 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1395983 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1395991 T4353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a7f7a5 name:ZooKeeperConnection Watcher:127.0.0.1:46568/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1395991 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1395992 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1395994 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1395997 T4351 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1395997 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1395997 T4348 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1395998 T4348 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1396000 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1396002 T4351 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1396104 T4344 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1396104 T4354 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1396109 T4289 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46568 46568
   [junit4]   2> 1396788 T4331 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46568 46568
   [junit4]   2> 1396790 T4289 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 1396791 T4289 oasc.Overseer.close Overseer (id=93051049418162180-127.0.0.1:46568_solr-n_0000000000) closing
   [junit4]   2> 1396792 T4348 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:46568_solr
   [junit4]   2> 1396794 T4355 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:90)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
   [junit4]   2> 
   [junit4]   2> 1396798 T4289 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 1396799 T4289 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1396799 T4356 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1396899 T4289 oasc.ZkTestServer.run start zk server on port:47757
   [junit4]   2> 1396900 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396902 T4362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9f6264 name:ZooKeeperConnection Watcher:127.0.0.1:47757 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396902 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396904 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396905 T4364 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@711de name:ZooKeeperConnection Watcher:127.0.0.1:47757 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396905 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396906 T4289 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1396908 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396909 T4366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1140030 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396909 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396909 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1396911 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396912 T4368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c2dfe8 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396912 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396912 T4289 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
   [junit4]   2> 1396913 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1396914 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1396915 T4289 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1396916 T4289 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47757_solr
   [junit4]   2> 1396916 T4289 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1396917 T4289 oasc.Overseer.start Overseer (id=93051049515089923-127.0.0.1:47757_solr-n_0000000000) starting
   [junit4]   2> 1396918 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1396919 T4289 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1396921 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1396922 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1396924 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1396925 T4289 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1396928 T4289 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1396928 T4370 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1396928 T4289 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1396930 T4369 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1396930 T4289 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1396931 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396932 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396933 T4373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f12f6 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396934 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396935 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396936 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 1396939 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396940 T4377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c675cc name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396940 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396942 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396943 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1396945 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396947 T4380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a6e364 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396947 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396948 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396950 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 1396953 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396955 T4383 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@944ca7 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396955 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396956 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396958 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 1396961 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396963 T4386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c5aa1 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396963 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396965 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396966 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 1396969 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396970 T4389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16f2b3d name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396971 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396971 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396973 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 1396976 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396977 T4392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1885cb7 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396977 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396979 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396980 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 1396983 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396984 T4395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9db15 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396984 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396985 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396986 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 1396990 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1396991 T4398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1406a91 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1396991 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1396992 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1396994 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 1397011 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397012 T4401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1468507 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397012 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397013 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397015 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 1397019 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397019 T4404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a103c name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397020 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397020 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397022 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 1397024 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397025 T4406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1061415 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397025 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397026 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397028 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 1397031 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397032 T4410 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c2f786 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397032 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397038 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397040 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 1397043 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397045 T4413 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b2bc4 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397045 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397046 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397047 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 1397058 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397058 T4416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e23507 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397059 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397059 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397060 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 1397064 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397065 T4419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1db36f2 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397065 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397066 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397067 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 1397071 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397072 T4422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cd8508 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397072 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397073 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397074 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 1397087 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397091 T4425 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@99ac7e name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397091 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397092 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397093 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 1397097 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397098 T4428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@108e2e7 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397098 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397098 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397107 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 1397113 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397113 T4431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e83a52 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397114 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397114 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397116 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 1397123 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397125 T4433 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c19e2 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397125 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397126 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397127 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 1397132 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397132 T4437 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127b68b name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397133 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397134 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397135 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 1397140 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397143 T4440 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e86ad2 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397144 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397144 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397145 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 1397150 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397151 T4443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188f957 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397152 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397152 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397154 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 1397158 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397160 T4446 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b16b8d name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397160 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397161 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397162 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 1397170 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397171 T4449 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d40b9e name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397171 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397172 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397173 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 1397177 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397178 T4452 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1324c0c name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397178 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397178 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397180 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 1397184 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397185 T4455 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17647bc name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397185 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397186 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397187 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 1397191 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397191 T4458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fba15c name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397192 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397192 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397194 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 1397198 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397199 T4461 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ced646 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397200 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397201 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397202 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 1397208 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397210 T4464 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14bbbbf name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397210 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397211 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397212 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 1397218 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397219 T4467 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f15e8d name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397219 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397220 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397221 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 1397227 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397229 T4470 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fef98e name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397229 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397230 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397231 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node32
   [junit4]   2> 1397238 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397238 T4473 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a6bf1 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397239 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397240 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397241 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node33
   [junit4]   2> 1397247 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397251 T4476 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e8abc0 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397251 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397252 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397254 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node34
   [junit4]   2> 1397260 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397261 T4479 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df8f26 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397261 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397262 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397264 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node35
   [junit4]   2> 1397270 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397272 T4482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15c253a name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397272 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397273 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397275 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node36
   [junit4]   2> 1397282 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397282 T4485 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94f403 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397283 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397284 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397285 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node37
   [junit4]   2> 1397293 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397294 T4488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ddf1fa name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397294 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397295 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397297 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node38
   [junit4]   2> 1397303 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397304 T4491 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10fef18 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397304 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397305 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397307 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node39
   [junit4]   2> 1397313 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397314 T4493 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a26636 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397314 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397315 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397317 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node40
   [junit4]   2> 1397324 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397325 T4496 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f5ac2d name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397325 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397326 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397328 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node41
   [junit4]   2> 1397334 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397334 T4500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73c168 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397335 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397335 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397337 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node42
   [junit4]   2> 1397342 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397343 T4503 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@349d51 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397343 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397344 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397345 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node43
   [junit4]   2> 1397351 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397351 T4506 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23dab7 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397352 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397353 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397355 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node44
   [junit4]   2> 1397360 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397361 T4509 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ee98da name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397361 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397362 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397364 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node45
   [junit4]   2> 1397370 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397370 T4512 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@128788e name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397371 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397372 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397374 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node46
   [junit4]   2> 1397380 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397381 T4515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@142bdaf name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397381 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397382 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397384 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node47
   [junit4]   2> 1397390 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397391 T4518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13e3192 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397391 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397392 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397394 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node48
   [junit4]   2> 1397399 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397401 T4521 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@af58a7 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397401 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397402 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397403 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node49
   [junit4]   2> 1397410 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397411 T4524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73a2b4 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397411 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397412 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397413 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node50
   [junit4]   2> 1397420 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397421 T4527 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed050 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397421 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397422 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397423 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node51
   [junit4]   2> 1397430 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397430 T4530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c1a2 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397431 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397431 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397433 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node52
   [junit4]   2> 1397441 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397442 T4533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e4de25 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397442 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397443 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397445 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node53
   [junit4]   2> 1397454 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397455 T4537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dbbe0f name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397456 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397456 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397457 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node54
   [junit4]   2> 1397465 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397466 T4540 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1376db4 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397467 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397467 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397468 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node55
   [junit4]   2> 1397475 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397476 T4543 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@196b261 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397476 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397477 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397478 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node56
   [junit4]   2> 1397485 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397486 T4546 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1814780 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397486 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397487 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397487 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node57
   [junit4]   2> 1397494 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397495 T4549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10abc39 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397495 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397496 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397497 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node58
   [junit4]   2> 1397505 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397505 T4552 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f62921 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397506 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397507 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397508 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node59
   [junit4]   2> 1397515 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397516 T4555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f78044 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397516 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397517 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397518 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node60
   [junit4]   2> 1397536 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397537 T4557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f36cbd name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397537 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397538 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397539 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node61
   [junit4]   2> 1397548 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397549 T4561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ef2ee name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397549 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397549 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397551 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node62
   [junit4]   2> 1397557 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397558 T4564 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e56525 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397558 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397559 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397560 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node63
   [junit4]   2> 1397569 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397570 T4567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11d399d name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397570 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397571 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397572 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node64
   [junit4]   2> 1397590 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397591 T4570 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@182347 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397591 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397592 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397594 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node65
   [junit4]   2> 1397604 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397606 T4573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16872a8 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397606 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397607 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397608 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node66
   [junit4]   2> 1397620 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397621 T4580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c5fe6d name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397622 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397623 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397625 T4289 oascc.SolrZkClient.makePath makePath: /live_nodes/node67
   [junit4]   2> 1397634 T4289 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1397634 T4584 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@baffb9 name:ZooKeeperConnection Watcher:127.0.0.1:47757/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1397634 T4289 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1397636 T4289 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1397637 T4289 oascc.SolrZkCl

[...truncated too long message...]

 state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1454271 T5123 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1454271 T5123 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1454272 T5130 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1454273 T5129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1454273 T5128 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1454770 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
   [junit4]   2> 1454773 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
   [junit4]   2> 1454775 T5130 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1454777 T5123 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1454777 T5123 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1454777 T5123 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1454778 T5130 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1454778 T5128 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1454778 T5129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1455275 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
   [junit4]   2> 1455280 T4289 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
   [junit4]   2> 1455283 T5130 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1455387 T5129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1455387 T5128 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1455436 T4289 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44827 44827
   [junit4]   2> 1456005 T5110 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44827 44827
   [junit4]   2> 1456006 T4289 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
   [junit4]   2> 1456006 T4289 oasc.Overseer.close Overseer (id=93051053129793539-127.0.0.1:44827_solr-n_0000000000) closing
   [junit4]   2> 1456007 T5123 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44827_solr
   [junit4]   2> 1456007 T5131 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:90)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
   [junit4]   2> 
   [junit4]   2> 1459008 T4289 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-core/test/J1/./solr.cloud.OverseerTest-57697C37D97D73C-001
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=uk_UA, timezone=America/Fort_Wayne
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_20 (32-bit)/cpus=8,threads=1,free=89330336,total=184868864
   [junit4]   2> NOTE: All tests run in this JVM: [TestPseudoReturnFields, LoggingHandlerTest, DirectSolrConnectionTest, BinaryUpdateRequestHandlerTest, TestSchemaVersionResource, TestCloudPivotFacet, TestSchemaResource, TestStressLucene, TestIndexSearcher, RequestHandlersTest, HdfsChaosMonkeySafeLeaderTest, LegacyHTMLStripCharFilterTest, CollectionsAPIDistributedZkTest, UniqFieldsUpdateProcessorFactoryTest, ChangedSchemaMergeTest, QueryParsingTest, AddSchemaFieldsUpdateProcessorFactoryTest, BJQParserTest, TestExpandComponent, CustomCollectionTest, TestCloudManagedSchemaConcurrent, ShardRoutingCustomTest, DistributedFacetPivotLongTailTest, TestSolr4Spatial2, TestManagedStopFilterFactory, TestCharFilters, TestRecovery, TestBadConfig, MergeStrategyTest, TestFunctionQuery, UUIDUpdateProcessorFallbackTest, AnalyticsQueryTest, NumericFieldsTest, TestOmitPositions, DistanceFunctionTest, DeleteInactiveReplicaTest, SimplePostToolTest, BlockCacheTest, TestShortCircuitedRequests, TestInfoStreamLogging, TestQuerySenderNoQuery, TestAnalyzeInfixSuggestions, TestBinaryField, ReplicationFactorTest, TestCollationField, TestReplicationHandlerBackup, TestReRankQParserPlugin, TestSolrQueryParser, TestQueryUtils, TestCopyFieldCollectionResource, HttpPartitionTest, TestMaxScoreQueryParser, TestPHPSerializedResponseWriter, TestFreeTextSuggestions, TestJoin, TestRandomFaceting, TestAtomicUpdateErrorCases, TestShardHandlerFactory, TestSolrQueryParserResource, ParsingFieldUpdateProcessorsTest, TestCoreContainer, OutputWriterTest, TestMergePolicyConfig, DeleteShardTest, SpatialFilterTest, TestPostingsSolrHighlighter, FullSolrCloudDistribCmdsTest, HardAutoCommitTest, SchemaVersionSpecificBehaviorTest, TestQueryTypes, BasicZkTest, TestSimpleQParserPlugin, TestPerFieldSimilarity, TestLFUCache, DistributedQueryComponentCustomSortTest, TestGroupingSearch, TestManagedSynonymFilterFactory, SolrTestCaseJ4Test, DirectUpdateHandlerOptimizeTest, TestDistributedMissingSort, ScriptEngineTest, MultiTermTest, TestSweetSpotSimilarityFactory, TestIBSimilarityFactory, HdfsRecoveryZkTest, DistributedQueryElevationComponentTest, IndexSchemaRuntimeFieldTest, URLClassifyProcessorTest, ZkSolrClientTest, SuggesterFSTTest, TestCursorMarkWithoutUniqueKey, TestFastWriter, DistributedDebugComponentTest, TestOverriddenPrefixQueryForCustomFieldType, TestFieldSortValues, UpdateRequestProcessorFactoryTest, TestDocumentBuilder, TestSchemaNameResource, CoreContainerCoreInitFailuresTest, TestSearcherReuse, SyncSliceTest, OverseerTest]
   [junit4] Completed on J1 in 67.23s, 9 tests, 1 error <<< FAILURES!

[...truncated 657 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/common-build.xml:494: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/lucene/common-build.xml:966: There were test failures: 421 suites, 1757 tests, 1 error, 25 ignored (18 assumptions)

Total time: 94 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC (asserts: false)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message