lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b15) - Build # 10440 - Still Failing!
Date Sun, 01 Jun 2014 12:37:29 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10440/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([4481675601E84A4B:4089E8A5134DA56A]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
	... 46 more




Build Log:
[...truncated 10865 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-4481675601E84A4B-001/init-core-data-001
   [junit4]   2> 293797 T1376 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 293797 T1376 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 293798 T1376 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 293801 T1376 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 293801 T1376 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 293802 T1377 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 293902 T1376 oasc.ZkTestServer.run start zk server on port:34004
   [junit4]   2> 293903 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 293996 T1383 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a755267 name:ZooKeeperConnection Watcher:127.0.0.1:34004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 293997 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 293997 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 293999 T1385 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d3edf55 name:ZooKeeperConnection Watcher:127.0.0.1:34004 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 293999 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 294002 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 294003 T1387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52f6f06c name:ZooKeeperConnection Watcher:127.0.0.1:34004 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294003 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 294004 T1376 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 294006 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 294008 T1376 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 294009 T1376 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 294010 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 294012 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 294013 T1389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6064e381 name:ZooKeeperConnection Watcher:127.0.0.1:34004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294013 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 294014 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 294016 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 294020 T1376 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34004_solr
   [junit4]   2> 294020 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 294022 T1376 oasc.Overseer.start Overseer (id=91856870291668995-127.0.0.1:34004_solr-n_0000000000) starting
   [junit4]   2> 294023 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 294026 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 294028 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 294030 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 294034 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 294037 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 294041 T1391 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 294042 T1390 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 294043 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294044 T1390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294044 T1390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 294044 T1390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 294048 T1389 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294049 T1383 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 294094 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294097 T1389 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294099 T1390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294100 T1390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 294100 T1390 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 294101 T1390 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 294105 T1389 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294208 T1383 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 294251 T1376 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:34004 34004
   [junit4]   2> 295081 T1377 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:34004 34004
   [junit4]   2> 295083 T1376 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   2> 295084 T1390 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:34004_solr
   [junit4]   2> 295086 T1392 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 295094 T1376 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 295095 T1376 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 295096 T1393 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 295196 T1376 oasc.ZkTestServer.run start zk server on port:42387
   [junit4]   2> 295197 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295201 T1399 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@526b122e name:ZooKeeperConnection Watcher:127.0.0.1:42387 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295201 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295204 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295208 T1401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ae70538 name:ZooKeeperConnection Watcher:127.0.0.1:42387 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295209 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295209 T1376 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 295215 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295217 T1403 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@84a91a7 name:ZooKeeperConnection Watcher:127.0.0.1:42387/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295217 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295218 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 295222 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295224 T1405 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20705e76 name:ZooKeeperConnection Watcher:127.0.0.1:42387/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295224 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295226 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 295229 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 295235 T1376 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42387_solr
   [junit4]   2> 295235 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 295238 T1376 oasc.Overseer.start Overseer (id=91856870376407043-127.0.0.1:42387_solr-n_0000000000) starting
   [junit4]   2> 295241 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 295248 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 295254 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 295258 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 295263 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 295269 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 295280 T1407 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 295280 T1376 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 295281 T1406 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 295284 T1376 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 295286 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 295292 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 295294 T1409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b226025 name:ZooKeeperConnection Watcher:127.0.0.1:42387/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295295 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 295296 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 295298 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 295304 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295307 T1406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 295308 T1406 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 295308 T1406 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 295309 T1406 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 295313 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295315 T1403 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 295315 T1409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 295804 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 295820 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 295827 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295831 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295831 T1406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 295838 T1406 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 295838 T1406 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 295839 T1406 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 295842 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295946 T1403 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 295946 T1409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 296331 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 296337 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 296341 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296341 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296341 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296342 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296343 T1406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296345 T1406 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 296345 T1406 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 296345 T1406 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 296347 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296449 T1409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 296449 T1403 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 296842 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 296855 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 296863 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296863 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296864 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296867 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296867 T1406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296874 T1406 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 296874 T1406 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 296874 T1406 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 296878 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296981 T1409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 296981 T1403 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 297367 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 297381 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297381 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297381 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297384 T1406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 297385 T1406 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 297385 T1406 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 297385 T1406 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 297391 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297625 T1403 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 297625 T1409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 297881 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 297896 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297896 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297896 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297899 T1406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 297900 T1406 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 297900 T1406 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 297901 T1406 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 297905 T1405 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 298008 T1403 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 298008 T1409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 298396 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 298405 T1376 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42387 42387
   [junit4]   2> 298973 T1393 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42387 42387
   [junit4]   2> 298974 T1376 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 298974 T1406 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:42387_solr
   [junit4]   2> 298975 T1410 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 298979 T1376 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 298979 T1376 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 298980 T1411 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 299080 T1376 oasc.ZkTestServer.run start zk server on port:50783
   [junit4]   2> 299081 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 299083 T1417 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8ae8a07 name:ZooKeeperConnection Watcher:127.0.0.1:50783/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 299083 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 299084 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 299085 T1419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c410459 name:ZooKeeperConnection Watcher:127.0.0.1:50783 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 299085 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 299087 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 299088 T1421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6717d886 name:ZooKeeperConnection Watcher:127.0.0.1:50783 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 299088 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 299089 T1376 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 299091 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 299093 T1376 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 299096 T1376 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 299098 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 299101 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 299104 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 299105 T1423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b429240 name:ZooKeeperConnection Watcher:127.0.0.1:50783/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 299106 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 299106 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 299108 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 299111 T1376 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50783_solr
   [junit4]   2> 299111 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 299113 T1376 oasc.Overseer.start Overseer (id=91856870630883331-127.0.0.1:50783_solr-n_0000000000) starting
   [junit4]   2> 299114 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 299117 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 299119 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 299121 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 299124 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 299129 T1425 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 299131 T1424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 299131 T1424 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 299132 T1424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 299133 T1417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 299135 T1424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 299136 T1417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 299138 T1424 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 299139 T1424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 299140 T1424 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 299141 T1423 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 299243 T1417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 299333 T1376 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50783 50783
   [junit4]   2> 299904 T1409 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 300100 T1409 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 300101 T1411 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50783 50783
   [junit4]   2> 300102 T1409 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 300103 T1376 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 300105 T1424 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50783_solr
   [junit4]   2> 300106 T1426 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 300114 T1376 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 300115 T1376 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 300116 T1427 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 300216 T1376 oasc.ZkTestServer.run start zk server on port:36307
   [junit4]   2> 300217 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300220 T1433 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20af43eb name:ZooKeeperConnection Watcher:127.0.0.1:36307 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300220 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300221 T1428 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x146575654930000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 300222 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300223 T1435 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7aeab93d name:ZooKeeperConnection Watcher:127.0.0.1:36307 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300224 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300224 T1376 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 300228 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300229 T1437 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6daa95d6 name:ZooKeeperConnection Watcher:127.0.0.1:36307/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300230 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300230 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 300232 T1376 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 300233 T1376 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 300235 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 300237 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300238 T1439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5db1fef2 name:ZooKeeperConnection Watcher:127.0.0.1:36307/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300239 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300240 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 300241 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 300244 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300245 T1441 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6109f85f name:ZooKeeperConnection Watcher:127.0.0.1:36307/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300245 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300247 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 300253 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 300257 T1376 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36307_solr
   [junit4]   2> 300257 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 300259 T1376 oasc.Overseer.start Overseer (id=91856870705397764-127.0.0.1:36307_solr-n_0000000000) starting
   [junit4]   2> 300261 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 300264 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 300266 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 300269 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 300272 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 300276 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 300281 T1443 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 300282 T1442 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 301284 T1441 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 301285 T1442 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 301286 T1442 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 301286 T1442 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 301286 T1442 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 301289 T1441 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 301290 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 301290 T1439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 301784 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 301862 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 301866 T1441 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 301867 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 301868 T1442 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 301871 T1441 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 301871 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 301873 T1442 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 301873 T1442 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 301877 T1441 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 301981 T1439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 301982 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 302283 T1443 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
   [junit4]   2> 303037 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 303068 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 303071 T1445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@545a4582 name:ZooKeeperConnection Watcher:127.0.0.1:36307/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 303071 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 303072 T1442 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:36307_solr
   [junit4]   2> 303073 T1446 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 303075 T1376 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36307_solr
   [junit4]   2> 303076 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 303078 T1376 oasc.Overseer.start Overseer (id=91856870705397765-127.0.0.1:36307_solr-n_0000000001) starting
   [junit4]   2> 303087 T1448 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 303088 T1447 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 303090 T1447 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 303090 T1447 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 303090 T1447 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 303092 T1445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 303093 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 303093 T1439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 303146 T1445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 303146 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 303148 T1447 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 303151 T1376 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91856870705397763-node1_core1-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:127)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 303152 T1439 oasc.LeaderElector.checkIfIamLeader WARN Our node is no longer in line to be leader
   [junit4]   2> 303154 T1447 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:177)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:174)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:631)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1066)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:346)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:242)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 303154 T1439 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 303154 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 303158 T1445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 303161 T1445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 303162 T1447 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 303164 T1445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 303266 T1439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 303266 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 304159 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 305162 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 306166 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 307174 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 308178 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 309181 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 310186 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 311189 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 312194 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 313197 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 314203 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 315209 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 316213 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 317217 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 318221 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 318232 T1376 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36307 36307
   [junit4]   2> 319106 T1448 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
   [junit4]   2> 323789 T1427 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36307 36307
   [junit4]   2> 323792 T1376 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 323792 T1447 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:36307_solr
   [junit4]   2> 323794 T1449 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=4481675601E84A4B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=nl -Dtests.timezone=Pacific/Tahiti -Dtests.file.encoding=UTF-8
   [junit4] ERROR   23.7s J0 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4481675601E84A4B:4089E8A5134DA56A]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
   [junit4]    > 	... 46 more
   [junit4]   2> 323818 T1376 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 323819 T1376 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 323819 T1450 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 323919 T1376 oasc.ZkTestServer.run start zk server on port:47503
   [junit4]   2> 323920 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 323924 T1456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23af07ac name:ZooKeeperConnection Watcher:127.0.0.1:47503/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323924 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 323926 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 323928 T1458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@774e7152 name:ZooKeeperConnection Watcher:127.0.0.1:47503 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323928 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 323932 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 323934 T1460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a90fc8 name:ZooKeeperConnection Watcher:127.0.0.1:47503 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323935 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 323935 T1376 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 323941 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 323947 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 323947 T1461 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 323950 T1463 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6728f481 name:ZooKeeperConnection Watcher:127.0.0.1:47503/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323950 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 323951 T1465 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31cc7610 name:ZooKeeperConnection Watcher:127.0.0.1:47503/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323951 T1461 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 323952 T1376 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 323953 T1461 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 323955 T1376 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 323957 T1461 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 323958 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 323963 T1461 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47503_solr
   [junit4]   2> 323964 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 323964 T1461 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 323968 T1461 oasc.Overseer.start Overseer (id=91856872258797572-127.0.0.1:47503_solr-n_0000000000) starting
   [junit4]   2> 323970 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 323972 T1461 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 323977 T1461 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 323982 T1461 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 323988 T1461 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 323993 T1461 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 323998 T1461 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 324007 T1461 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 324007 T1467 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 324009 T1461 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 324011 T1469 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a2ac8e name:ZooKeeperConnection Watcher:127.0.0.1:47503/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 324011 T1461 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 324012 T1467 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=91856872258797572-127.0.0.1:47503_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 324013 T1467 oasc.SolrException.log ERROR :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 		at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:98)
   [junit4]   2> 		at org.apache.solr.cloud.DistributedQueue.getTailId(DistributedQueue.java:450)
   [junit4]   2> 		at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:241)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 324014 T1467 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:408)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:256)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 324017 T1461 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47503_solr
   [junit4]   2> 324017 T1461 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 324021 T1461 oasc.Overseer.start Overseer (id=91856872258797573-127.0.0.1:47503_solr-n_0000000001) starting
   [junit4]   2> 324040 T1471 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 324042 T1470 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 324047 T1470 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 324048 T1470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 324048 T1470 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 324048 T1470 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 324053 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324055 T1463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 324476 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 324490 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 324501 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324505 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324505 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 324505 T1470 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 324513 T1470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 324513 T1470 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 324515 T1376 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 324517 T1473 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@618d4438 name:ZooKeeperConnection Watcher:127.0.0.1:47503/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 324517 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 324518 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324520 T1376 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 324525 T1376 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 324532 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324532 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 324534 T1470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 324534 T1470 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 324539 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324542 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 324641 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 324642 T1463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 324744 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 324749 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324749 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324750 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324752 T1470 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 324753 T1470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 324754 T1470 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 324754 T1470 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 324759 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 324862 T1463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 324862 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 325249 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 325263 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 325263 T1473 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 325272 T1469 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4

[...truncated too long message...]

anager@127208c7 name:ZooKeeperConnection Watcher:127.0.0.1:52026/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 352496 T1376 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 352497 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 352498 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 352500 T1376 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52026_solr
   [junit4]   2> 352500 T1376 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 352501 T1376 oasc.Overseer.start Overseer (id=91856874129719300-127.0.0.1:52026_solr-n_0000000000) starting
   [junit4]   2> 352502 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 352503 T1376 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 352505 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 352509 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 352511 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 352514 T1376 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 352517 T1947 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 352518 T1946 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 352519 T1946 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 352520 T1946 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 352520 T1946 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 352521 T1946 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard6
   [junit4]   2> 352523 T1945 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 352525 T1937 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 352525 T1943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 353018 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard6/election
   [junit4]   2> 353024 T1376 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard6
   [junit4]   2> 353027 T1945 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 353027 T1376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 353028 T1946 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 353028 T1946 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work/qn-
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:313)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:297)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:255)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 353029 T1946 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52026_solr
   [junit4]   2> 353029 T1948 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 353031 T1376 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52026 52026
   [junit4]   2> 353580 T1931 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52026 52026
   [junit4]   2> 353583 T1376 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   2> 353586 T1376 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 353587 T1376 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 356587 T1376 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-4481675601E84A4B-001
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=nl, timezone=Pacific/Tahiti
   [junit4]   2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=286738456,total=369098752
   [junit4]   2> NOTE: All tests run in this JVM: [UUIDFieldTest, QueryEqualityTest, RAMDirectoryFactoryTest, EnumFieldTest, DistributedExpandComponentTest, ModifyConfFileTest, TestShortCircuitedRequests, CopyFieldTest, TestFastLRUCache, TestGroupingSearch, PrimitiveFieldTypeTest, TermsComponentTest, InfoHandlerTest, ClusterStateUpdateTest, TestSchemaVersionResource, TestSolrXmlPersistor, TestDistribDocBasedVersion, TestOrdValues, CacheHeaderTest, HdfsRecoveryZkTest, TestPerFieldSimilarity, TestInfoStreamLogging, BadComponentTest, ParsingFieldUpdateProcessorsTest, TestSurroundQueryParser, SampleTest, DistributedSpellCheckComponentTest, TestFieldSortValues, TestMergePolicyConfig, ShowFileRequestHandlerTest, SchemaVersionSpecificBehaviorTest, TestFreeTextSuggestions, TestWriterPerf, QueryParsingTest, AliasIntegrationTest, TestDynamicFieldCollectionResource, LukeRequestHandlerTest, DistributedTermsComponentTest, TestComplexPhraseQParserPlugin, TestRandomMergePolicy, ReturnFieldsTest, SearchHandlerTest, ReplicationFactorTest, TestSort, FieldMutatingUpdateProcessorTest, TestRealTimeGet, URLClassifyProcessorTest, SpatialFilterTest, FileBasedSpellCheckerTest, TimeZoneUtilsTest, TestHighFrequencyDictionaryFactory, TestExtendedDismaxParser, MBeansHandlerTest, TestFieldCollectionResource, OverseerTest]
   [junit4] Completed on J0 in 62.82s, 9 tests, 1 error <<< FAILURES!

[...truncated 937 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 error, 36 ignored (15 assumptions)

Total time: 66 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message