lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1737 - Failure!
Date Mon, 28 Jul 2014 09:44:42 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1737/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([CCBC2926C68051AC:C8B4A6D5D425BE8D]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131)
	... 47 more




Build Log:
[...truncated 11508 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-CCBC2926C68051AC-001/init-core-data-001
   [junit4]   2> 3841469 T7732 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 3841469 T7732 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 3841470 T7732 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 3841475 T7732 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 3841477 T7732 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3841478 T7733 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3841579 T7732 oasc.ZkTestServer.run start zk server on port:51636
   [junit4]   2> 3841582 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3841596 T7739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@611311d name:ZooKeeperConnection Watcher:127.0.0.1:51636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3841598 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3841602 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3841609 T7741 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17300835 name:ZooKeeperConnection Watcher:127.0.0.1:51636 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3841610 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3841618 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3841622 T7743 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24966b4a name:ZooKeeperConnection Watcher:127.0.0.1:51636 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3841622 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3841623 T7732 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3841631 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3841642 T7732 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3841649 T7732 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3841652 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3841662 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3841666 T7745 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d0561d9 name:ZooKeeperConnection Watcher:127.0.0.1:51636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3841666 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3841673 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3841687 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 3841699 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3841703 T7747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b10f623 name:ZooKeeperConnection Watcher:127.0.0.1:51636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3841704 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3841706 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3841714 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3841727 T7732 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51636_solr
   [junit4]   2> 3841728 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3841735 T7732 oasc.Overseer.start Overseer (id=92178858700177412-127.0.0.1:51636_solr-n_0000000000) starting
   [junit4]   2> 3841741 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3841753 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3841764 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3841773 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3841783 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3841795 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3841812 T7749 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3841816 T7748 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3841823 T7748 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3841826 T7748 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3841826 T7748 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 3841826 T7748 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 3841833 T7747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3841835 T7745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3841835 T7739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3842318 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 3842354 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 3842371 T7747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3842372 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3842374 T7748 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3842387 T7732 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51636 51636
   [junit4]   2> 3842398 T7733 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51636 51636
   [junit4]   2> 3842619 T7732 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   2> 3842621 T7748 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51636_solr
   [junit4]   2> 3842623 T7750 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 3842641 T7732 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 3842651 T7732 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3842671 T7751 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3842760 T7732 oasc.ZkTestServer.run start zk server on port:51642
   [junit4]   2> 3842762 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3842770 T7757 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64a1af1 name:ZooKeeperConnection Watcher:127.0.0.1:51642/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3842770 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3842774 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3842780 T7759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bfad2b1 name:ZooKeeperConnection Watcher:127.0.0.1:51642 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3842781 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3842788 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3842791 T7761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c192f93 name:ZooKeeperConnection Watcher:127.0.0.1:51642 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3842791 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3842791 T7732 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3842804 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3842812 T7732 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3842818 T7732 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3842823 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3842834 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3842840 T7763 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a02ab9e name:ZooKeeperConnection Watcher:127.0.0.1:51642/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3842840 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3842846 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3842854 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 3842866 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3842869 T7765 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@438a163c name:ZooKeeperConnection Watcher:127.0.0.1:51642/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3842869 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3842873 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3842880 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3842894 T7732 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51642_solr
   [junit4]   2> 3842894 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3842901 T7732 oasc.Overseer.start Overseer (id=92178858778427396-127.0.0.1:51642_solr-n_0000000000) starting
   [junit4]   2> 3842910 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3842921 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3842932 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3842940 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3842950 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3842961 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3842980 T7767 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3842983 T7766 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3842989 T7766 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3842991 T7766 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3842991 T7766 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 3842991 T7766 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3842998 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3842999 T7763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3842999 T7757 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3843485 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3843520 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3843535 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3843536 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3843539 T7766 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3843548 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3843549 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3843552 T7769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43721cf5 name:ZooKeeperConnection Watcher:127.0.0.1:51642/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3843552 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3843556 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3843565 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 3843575 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3843575 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3843578 T7766 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3843578 T7766 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3843585 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3843593 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3843606 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3843615 T7765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3843722 T7769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3843722 T7757 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3843727 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3843738 T7732 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51642 51642
   [junit4]   2> 3843774 T7751 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51642 51642
   [junit4]   2> 3844072 T7732 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 3844077 T7766 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51642_solr
   [junit4]   2> 3844081 T7770 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 3844089 T7732 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 3844090 T7732 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3844091 T7771 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3844193 T7732 oasc.ZkTestServer.run start zk server on port:51649
   [junit4]   2> 3844196 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3844212 T7777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7323b98d name:ZooKeeperConnection Watcher:127.0.0.1:51649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3844213 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3844221 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3844226 T7779 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1073d45f name:ZooKeeperConnection Watcher:127.0.0.1:51649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3844227 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3844227 T7732 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3844236 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3844241 T7781 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e75b45d name:ZooKeeperConnection Watcher:127.0.0.1:51649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3844242 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3844242 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3844250 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3844254 T7783 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57c991e1 name:ZooKeeperConnection Watcher:127.0.0.1:51649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3844254 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3844257 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3844264 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3844278 T7732 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51649_solr
   [junit4]   2> 3844278 T7732 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3844284 T7732 oasc.Overseer.start Overseer (id=92178858871422979-127.0.0.1:51649_solr-n_0000000000) starting
   [junit4]   2> 3844289 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3844300 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3844313 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3844323 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3844332 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3844345 T7732 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3844367 T7785 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3844369 T7732 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3844371 T7784 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3844376 T7732 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3844384 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3844395 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3844399 T7787 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3028b881 name:ZooKeeperConnection Watcher:127.0.0.1:51649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3844399 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3844404 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3844413 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 3844425 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3844430 T7784 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3844432 T7784 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 3844432 T7784 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 3844432 T7784 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 3844440 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3844442 T7787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3844443 T7781 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3844927 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 3844957 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 3844978 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3844983 T7784 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3844983 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3844994 T7784 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 3844994 T7784 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3844994 T7784 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 3845001 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3845109 T7781 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3845109 T7787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3845484 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 3845517 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 3845535 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3845536 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3845536 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3845542 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3845542 T7784 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3845557 T7784 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 3845557 T7784 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3845558 T7784 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3845566 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3845675 T7787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3845676 T7781 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3846044 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3846073 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3846090 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846090 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846091 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846095 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846095 T7784 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3846111 T7784 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 3846111 T7784 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3846111 T7784 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 3846119 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846227 T7781 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3846227 T7787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3846596 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 3846630 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846630 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846630 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846635 T7784 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3846638 T7784 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 3846639 T7784 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3846639 T7784 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 3846647 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3846756 T7787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3846757 T7781 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3847131 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 3847168 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3847169 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3847169 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3847174 T7784 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3847177 T7784 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 3847178 T7784 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3847178 T7784 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3847188 T7783 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3847296 T7781 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3847297 T7787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3847670 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3847709 T7732 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51649 51649
   [junit4]   2> 3847720 T7771 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51649 51649
   [junit4]   2> 3847993 T7732 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 3847994 T7784 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51649_solr
   [junit4]   2> 3847997 T7788 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 3848006 T7732 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 3848008 T7732 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3848009 T7789 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3848109 T7732 oasc.ZkTestServer.run start zk server on port:51655
   [junit4]   2> 3848112 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848126 T7795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66191798 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848126 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848128 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848132 T7797 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3eb8a03c name:ZooKeeperConnection Watcher:127.0.0.1:51655 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848133 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848143 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848148 T7799 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51925ad name:ZooKeeperConnection Watcher:127.0.0.1:51655 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848149 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848149 T7732 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3848162 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3848173 T7800 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848174 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848178 T7802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@196c8faa name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848179 T7800 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848179 T7804 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f927996 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848180 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848181 T7800 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3848181 T7732 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3848185 T7800 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3848187 T7732 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3848191 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3848194 T7800 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51655_solr
   [junit4]   2> 3848194 T7800 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3848196 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 3848199 T7800 oasc.Overseer.start Overseer (id=92178859128127491-127.0.0.1:51655_solr-n_0000000000) starting
   [junit4]   2> 3848203 T7800 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3848204 T7732 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3848209 T7800 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3848218 T7800 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3848226 T7800 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3848233 T7800 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3848245 T7800 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3848261 T7800 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 3848262 T7806 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3848267 T7800 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848269 T7808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b3cc608 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848269 T7800 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848270 T7806 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92178859128127491-127.0.0.1:51655_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 3848272 T7806 oasc.SolrException.log ERROR :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 		at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:99)
   [junit4]   2> 		at org.apache.solr.cloud.DistributedQueue.getTailId(DistributedQueue.java:483)
   [junit4]   2> 		at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:241)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 3848273 T7806 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:412)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:256)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3848279 T7800 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51655_solr
   [junit4]   2> 3848279 T7800 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3848285 T7800 oasc.Overseer.start Overseer (id=92178859128127493-127.0.0.1:51655_solr-n_0000000001) starting
   [junit4]   2> 3848312 T7810 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3848316 T7809 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3848324 T7809 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3848325 T7809 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3848325 T7809 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 3848326 T7809 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3848333 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3848335 T7804 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3848710 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3848737 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3848751 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3848755 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3848755 T7809 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3848755 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3848765 T7809 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3848767 T7809 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3848774 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3848774 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3848778 T7812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ba813ac name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3848778 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3848782 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3848790 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 3848800 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3848800 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3848803 T7809 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3848803 T7809 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3848810 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3848819 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3848920 T7812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3848921 T7804 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3849025 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849039 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849039 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849040 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849046 T7809 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849048 T7809 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 3849049 T7809 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 3849049 T7809 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3849055 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849166 T7812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3849166 T7804 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3849206 T7787 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 3849208 T7787 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 3849209 T7787 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 3849540 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3849574 T7804 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 3849575 T7804 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 3849577 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849580 T7812 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3849594 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849595 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849595 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849601 T7809 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849610 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849718 T7812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3849785 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849791 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3849797 T7814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4264cda1 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3849797 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3849802 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3849809 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 3849824 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849824 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3849824 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849826 T7808 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849831 T7800 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 3849837 T7800 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3849843 T7816 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f1de0aa name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3849844 T7800 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3849846 T7809 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51655_solr
   [junit4]   2> 3849851 T7817 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 3849862 T7800 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51655_solr
   [junit4]   2> 3849862 T7800 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3849862 T7814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3849864 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3849866 T7800 oasc.Overseer.start Overseer (id=92178859128127496-127.0.0.1:51655_solr-n_0000000002) starting
   [junit4]   2> 3849881 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3849883 T7819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5beaffd9 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3849884 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3849887 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3849891 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 3849893 T7821 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3849897 T7820 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3849901 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849901 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3849902 T7820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849904 T7820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3849904 T7820 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3849911 T7814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3849911 T7819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3849916 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849921 T7820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3849922 T7820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3849923 T7820 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3849928 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849935 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3849938 T7820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3849938 T7820 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3849945 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850051 T7819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3850053 T7814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3850122 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850134 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850134 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850134 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850134 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3850138 T7820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850140 T7820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 3850140 T7820 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3850146 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850155 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850156 T7819 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3850170 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850180 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850286 T7819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3850360 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850364 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3850370 T7823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@190d7407 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3850370 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3850374 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3850381 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 3850391 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3850391 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850392 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850392 T7816 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850400 T7800 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 3850400 T7820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850405 T7800 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3850410 T7825 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ff42531 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3850411 T7800 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3850412 T7820 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51655_solr
   [junit4]   2> 3850414 T7826 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 3850423 T7800 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51655_solr
   [junit4]   2> 3850423 T7800 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3850425 T7823 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3850428 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3850429 T7800 oasc.Overseer.start Overseer (id=92178859128127499-127.0.0.1:51655_solr-n_0000000003) starting
   [junit4]   2> 3850444 T7732 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3850447 T7828 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44ce6100 name:ZooKeeperConnection Watcher:127.0.0.1:51655/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3850448 T7732 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3850451 T7732 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3850456 T7830 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3850458 T7829 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3850458 T7732 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 3850465 T7829 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850466 T7829 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3850467 T7829 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3850468 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850469 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3850474 T7823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3850474 T7828 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3850484 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850485 T7829 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850486 T7829 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3850487 T7829 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3850493 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850502 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850505 T7829 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 3850505 T7829 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 3850511 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850621 T7823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3850621 T7828 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3850690 T7732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850706 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850707 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850707 T7825 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3850706 T7732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3850714 T7829 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3850716 T7829 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"

[...truncated too long message...]

t org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 3891312 T8082 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders/shard12
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136)
   [junit4]   2> 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131)
   [junit4]   2> 	... 6 more
   [junit4]   2> 
   [junit4]   2> 3891314 T8082 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3891328 T8088 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3893230 T7732 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-CCBC2926C68051AC-001
   [junit4]   2> NOTE: test params are: codec=Lucene49: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ca, timezone=America/Guayaquil
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=2,threads=1,free=134107312,total=375390208
   [junit4]   2> NOTE: All tests run in this JVM: [TestDefaultSearchFieldResource, SampleTest, TestUpdate, ShardRoutingTest, TriLevelCompositeIdRoutingTest, TestLuceneMatchVersion, MergeStrategyTest, DirectSolrConnectionTest, CurrencyFieldOpenExchangeTest, TestStressVersions, PreAnalyzedFieldTest, TestLeaderElectionZkExpiry, CurrencyFieldXmlFileTest, SuggesterFSTTest, AnalysisAfterCoreReloadTest, TestSolrXmlPersistor, SolrCmdDistributorTest, TestPerFieldSimilarity, TestCollectionAPI, SolrRequestParserTest, ResourceLoaderTest, SpellPossibilityIteratorTest, SolrXmlInZkTest, TestNonNRTOpen, CSVRequestHandlerTest, DistributedTermsComponentTest, TestZkChroot, TestOverriddenPrefixQueryForCustomFieldType, TestJmxIntegration, TestSolrQueryParserDefaultOperatorResource, TestRandomFaceting, SolrCoreCheckLockOnStartupTest, TestDocBasedVersionConstraints, TestSort, BadComponentTest, TestInitQParser, TestPartialUpdateDeduplication, TestSolrQueryParserResource, TestIBSimilarityFactory, TestArbitraryIndexDir, ResponseHeaderTest, TestUniqueKeyFieldResource, BlockDirectoryTest, InfoHandlerTest, TestDistributedGrouping, SuggestComponentTest, HighlighterConfigTest, TestRecovery, OpenExchangeRatesOrgProviderTest, NoCacheHeaderTest, TestRequestStatusCollectionAPI, TestRemoteStreaming, ExpressionTest, HdfsSyncSliceTest, DistanceFunctionTest, MultiThreadedOCPTest, TestSystemIdResolver, DeleteInactiveReplicaTest, TestLMJelinekMercerSimilarityFactory, ModifyConfFileTest, URLClassifyProcessorTest, WordBreakSolrSpellCheckerTest, SoftAutoCommitTest, TermVectorComponentDistributedTest, UUIDFieldTest, TestFunctionQuery, DeleteShardTest, TestTrie, PolyFieldTest, SystemInfoHandlerTest, TestCodecSupport, DistributedQueryComponentCustomSortTest, MigrateRouteKeyTest, CoreMergeIndexesAdminHandlerTest, TestDFRSimilarityFactory, StandardRequestHandlerTest, AddBlockUpdateTest, TestSolr4Spatial, TestCloudSchemaless, DocValuesMultiTest, TestSurroundQueryParser, DirectUpdateHandlerTest, TestLRUCache, TestSolrXml, PrimitiveFieldTypeTest, TestRestManager, TestXIncludeConfig, SSLMigrationTest, TestIndexingPerformance, HdfsUnloadDistributedZkTest, TestCoreDiscovery, TestSolrCoreProperties, QueryFacetTest, TestWriterPerf, TermsComponentTest, DocValuesTest, QueryResultKeyTest, TestDistribDocBasedVersion, FastVectorHighlighterTest, FieldAnalysisRequestHandlerTest, TestBadConfig, TestReloadAndDeleteDocs, TestNonDefinedSimilarityFactory, MoreLikeThisHandlerTest, DistributedQueryElevationComponentTest, FieldMutatingUpdateProcessorTest, IndexSchemaRuntimeFieldTest, ReplicationFactorTest, ZkNodePropsTest, DistributedIntervalFacetingTest, TestShardHandlerFactory, TestSchemaNameResource, TestPseudoReturnFields, FileBasedSpellCheckerTest, ChaosMonkeyNothingIsSafeTest, TestSolrJ, TestSearchPerf, AnalysisErrorHandlingTest, TestManagedResourceStorage, TestManagedSchemaFieldResource, UpdateParamsTest, ChaosMonkeySafeLeaderTest, ShardSplitTest, TestCustomSort, TestReload, TestBM25SimilarityFactory, OverseerCollectionProcessorTest, SolrInfoMBeanTest, AlternateDirectoryTest, TestElisionMultitermQuery, HdfsLockFactoryTest, TestSerializedLuceneMatchVersion, UnloadDistributedZkTest, TestReRankQParserPlugin, RequiredFieldsTest, QueryEqualityTest, NumericFieldsTest, TestAddFieldRealTimeGet, IndexBasedSpellCheckerTest, TestManagedStopFilterFactory, HdfsBasicDistributedZkTest, TestJoin, RangeFacetTest, TestQueryTypes, DistribDocExpirationUpdateProcessorTest, TestPostingsSolrHighlighter, DateFieldTest, DebugComponentTest, TestAnalyzeInfixSuggestions, TestFoldingMultitermQuery, TestStandardQParsers, BJQParserTest, SpatialFilterTest, TestDistributedMissingSort, TestHighFrequencyDictionaryFactory, TestTolerantSearch, TestPHPSerializedResponseWriter, QueryElevationComponentTest, DistributedDebugComponentTest, BasicFunctionalityTest, CoreAdminHandlerTest, NotRequiredUniqueKeyTest, TestExpandComponent, ExternalFileFieldSortTest, CircularListTest, SuggesterTSTTest, TestModifyConfFiles, ClusterStateUpdateTest, SolrPluginUtilsTest, HdfsRecoveryZkTest, TestRandomMergePolicy, TestCSVLoader, TestNRTOpen, SpellingQueryConverterTest, DOMUtilTest, HdfsCollectionsAPIDistributedZkTest, TestBlendedInfixSuggestions, PrimUtilsTest, TestSearcherReuse, DistribCursorPagingTest, HdfsDirectoryTest, TestQuerySenderNoQuery, TestFuzzyAnalyzedSuggestions, TimeZoneUtilsTest, StatelessScriptUpdateProcessorFactoryTest, TestCollapseQParserPlugin, TestImplicitCoreProperties, TestDistributedSearch, DirectSolrSpellCheckerTest, HighlighterMaxOffsetTest, BasicZkTest, OverseerStatusTest, TestQuerySenderListener, TestReversedWildcardFilterFactory, SchemaVersionSpecificBehaviorTest, TestBinaryResponseWriter, CachingDirectoryFactoryTest, BasicDistributedZk2Test, TestSolrDeletionPolicy2, TestQueryUtils, AnalyticsQueryTest, TestFieldTypeCollectionResource, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest]
   [junit4] Completed in 51.83s, 9 tests, 1 error <<< FAILURES!

[...truncated 632 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1298: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:922: There were test failures: 421 suites, 1751 tests, 1 error, 44 ignored (17 assumptions)

Total time: 157 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message