lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Windows (64bit/jdk1.7.0_55) - Build # 3913 - Failure!
Date Tue, 29 Apr 2014 11:52:08 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3913/
Java: 64bit/jdk1.7.0_55 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([43938CF4049E4FF0:479B0307163BA0D1]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
	... 46 more
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
	... 47 more




Build Log:
[...truncated 10881 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solr.cloud.OverseerTest-43938CF4049E4FF0-001\init-core-data-001
   [junit4]   2> 423971 T1245 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 423971 T1245 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 423971 T1245 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 423974 T1245 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 423977 T1245 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 423977 T1246 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 424103 T1245 oasc.ZkTestServer.run start zk server on port:60286
   [junit4]   2> 424106 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 424116 T1252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7215d5a9 name:ZooKeeperConnection Watcher:127.0.0.1:60286/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 424117 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 424120 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 424125 T1254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ce6448 name:ZooKeeperConnection Watcher:127.0.0.1:60286 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 424125 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 424132 T1247 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18567d0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 424135 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 424141 T1256 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f79ec4 name:ZooKeeperConnection Watcher:127.0.0.1:60286 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 424141 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 424141 T1245 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 424153 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 424153 T1247 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18567d0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 424165 T1245 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 424179 T1245 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 424199 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 424210 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 424226 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 424230 T1258 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d25c768 name:ZooKeeperConnection Watcher:127.0.0.1:60286/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 424230 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 424232 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 424241 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 424254 T1245 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60286_solr
   [junit4]   2> 424256 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 424263 T1245 oasc.Overseer.start Overseer (id=91669687472816131-127.0.0.1:60286_solr-n_0000000000) starting
   [junit4]   2> 424270 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 424288 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 424299 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 424311 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 424324 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 424344 T1260 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 424346 T1260 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60286_solr
   [junit4]   2> 424351 T1259 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 424352 T1259 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 424354 T1259 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 424356 T1252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 424365 T1259 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 424367 T1252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 424372 T1259 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 424377 T1259 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 424378 T1259 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 424384 T1258 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 424387 T1252 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 424478 T1247 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18567d0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 424483 T1247 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18567d0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 424483 T1245 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60286 60286
   [junit4]   2> 427308 T1246 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60286 60286
   [junit4]   2> 427311 T1245 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 427311 T1259 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60286_solr
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 427314 T1261 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 427321 T1245 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem
   [junit4]   2> 427325 T1245 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 427326 T1262 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 427451 T1245 oasc.ZkTestServer.run start zk server on port:60301
   [junit4]   2> 427452 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427462 T1268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@634284f3 name:ZooKeeperConnection Watcher:127.0.0.1:60301 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427463 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427467 T1263 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18638b0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 427469 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427473 T1270 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b81bc76 name:ZooKeeperConnection Watcher:127.0.0.1:60301 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427473 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427473 T1245 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 427484 T1263 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18638b0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 427486 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427489 T1272 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@125973c name:ZooKeeperConnection Watcher:127.0.0.1:60301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427490 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427490 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 427497 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427501 T1274 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bb60358 name:ZooKeeperConnection Watcher:127.0.0.1:60301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427502 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427503 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 427510 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 427519 T1245 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60301_solr
   [junit4]   2> 427521 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 427524 T1245 oasc.Overseer.start Overseer (id=91669687691837443-127.0.0.1:60301_solr-n_0000000000) starting
   [junit4]   2> 427529 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 427536 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 427544 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 427550 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 427558 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 427567 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 427582 T1276 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 427583 T1245 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 427584 T1276 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60301_solr
   [junit4]   2> 427585 T1275 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 427590 T1245 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 427595 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 427603 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427606 T1278 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39835e11 name:ZooKeeperConnection Watcher:127.0.0.1:60301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427606 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427610 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 427616 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 427623 T1278 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 427623 T1272 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 427629 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 427633 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 427634 T1275 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 427634 T1275 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 427634 T1275 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 427642 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 427644 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 427645 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 428256 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 428278 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 428291 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428296 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428296 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 428306 T1275 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 428306 T1275 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 428306 T1275 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 428312 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428443 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 428444 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 428906 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 428922 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 428934 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428934 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428934 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428939 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 428941 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428950 T1275 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 428951 T1275 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 428951 T1275 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 428956 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429075 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 429076 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 429506 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 429524 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 429538 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429538 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429538 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429541 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 429549 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429667 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 429668 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 429715 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429716 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429716 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429719 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429720 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 429723 T1275 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":"",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 429729 T1275 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2> 	  "operation":"deletecore",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":""}
   [junit4]   2> 429735 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429736 T1275 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 429737 T1275 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3]
   [junit4]   2> 429737 T1275 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 429741 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429860 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 429860 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 430289 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election
   [junit4]   2> 430327 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2
   [junit4]   2> 430350 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430350 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430352 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430355 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430356 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 430365 T1275 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 430365 T1275 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 430366 T1275 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 430370 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430488 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 430488 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 430920 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
   [junit4]   2> 430939 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
   [junit4]   2> 430952 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430952 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430952 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430956 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 430957 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 430967 T1275 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 430968 T1275 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 430968 T1275 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 430974 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 431093 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 431094 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 431521 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
   [junit4]   2> 431540 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
   [junit4]   2> 431552 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 431553 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 431553 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 431556 T1275 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 431564 T1274 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 431683 T1272 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 431684 T1278 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 431730 T1278 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 431734 T1263 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18638b0004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 431737 T1263 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18638b0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 431738 T1245 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60301 60301
   [junit4]   2> 434280 T1262 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60301 60301
   [junit4]   2> 434282 T1245 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
   [junit4]   2> 434283 T1275 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60301_solr
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 434284 T1279 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 434290 T1245 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 434292 T1245 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 434293 T1280 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 434405 T1245 oasc.ZkTestServer.run start zk server on port:60319
   [junit4]   2> 434409 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 434415 T1286 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@191f5509 name:ZooKeeperConnection Watcher:127.0.0.1:60319/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 434416 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 434418 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 434421 T1288 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c0009c0 name:ZooKeeperConnection Watcher:127.0.0.1:60319 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 434421 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 434425 T1281 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad187ec30001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 434426 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 434430 T1290 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22335a6b name:ZooKeeperConnection Watcher:127.0.0.1:60319 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 434431 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 434431 T1245 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 434437 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 434437 T1281 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad187ec30002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 434444 T1245 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 434449 T1245 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 434453 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 434458 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 434462 T1292 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f456852 name:ZooKeeperConnection Watcher:127.0.0.1:60319/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 434463 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 434465 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 434469 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 434475 T1292 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 434476 T1286 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 434477 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 434480 T1294 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@243f9178 name:ZooKeeperConnection Watcher:127.0.0.1:60319/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 434481 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 434483 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 434488 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 434497 T1245 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60319_solr
   [junit4]   2> 434497 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 434502 T1245 oasc.Overseer.start Overseer (id=91669688148492292-127.0.0.1:60319_solr-n_0000000000) starting
   [junit4]   2> 434508 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 434514 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 434521 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 434528 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 434534 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 434542 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 434556 T1296 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 434557 T1296 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60319_solr
   [junit4]   2> 434561 T1295 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 434565 T1295 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 434566 T1295 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 434567 T1295 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 434567 T1295 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard6
   [junit4]   2> 434573 T1294 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 434576 T1286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 434576 T1292 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 435123 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard6/election
   [junit4]   2> 435142 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard6
   [junit4]   2> 435153 T1294 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 435154 T1245 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 435157 T1295 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 435160 T1281 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad187ec30004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 435162 T1292 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 435163 T1286 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 435166 T1281 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad187ec30000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 435168 T1245 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60319 60319
   [junit4]   2> 436848 T1295 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:239)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:422)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 437705 T1295 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60319_solr
   [junit4]   2> 437715 T1297 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 437718 T1280 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60319 60319
   [junit4]   2> 437725 T1245 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 437739 T1245 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 437742 T1245 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 437745 T1298 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 437857 T1245 oasc.ZkTestServer.run start zk server on port:60337
   [junit4]   2> 437860 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 437875 T1304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3140538f name:ZooKeeperConnection Watcher:127.0.0.1:60337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 437875 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 437880 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 437886 T1306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c47c5 name:ZooKeeperConnection Watcher:127.0.0.1:60337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 437886 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 437899 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 437905 T1308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@558bf6b8 name:ZooKeeperConnection Watcher:127.0.0.1:60337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 437905 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 437906 T1245 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 437922 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 437923 T1299 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad188c420002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 437935 T1245 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 437947 T1245 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 437955 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 437969 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 437976 T1310 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15b8b142 name:ZooKeeperConnection Watcher:127.0.0.1:60337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 437977 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 437980 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 437992 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 438014 T1245 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60337_solr
   [junit4]   2> 438017 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 438026 T1245 oasc.Overseer.start Overseer (id=91669688374919171-127.0.0.1:60337_solr-n_0000000000) starting
   [junit4]   2> 438039 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 438055 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 438073 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 438089 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 438102 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 438120 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 438147 T1312 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 438150 T1312 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60337_solr
   [junit4]   2> 438152 T1311 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 438156 T1245 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 438162 T1311 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 438163 T1311 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 438164 T1311 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 438176 T1310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 438181 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 438220 T1245 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 438231 T1310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 438238 T1311 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 438240 T1311 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 438241 T1311 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 438241 T1311 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 438252 T1310 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 438373 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 438408 T1299 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad188c420000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 438414 T1299 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad188c420003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 438416 T1245 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60337 60337
   [junit4]   2> 440868 T1298 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60337 60337
   [junit4]   2> 440871 T1245 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   2> 440872 T1311 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60337_solr
   [junit4]   2> 440875 T1313 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 440888 T1245 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 440892 T1245 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 440893 T1314 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 440995 T1245 oasc.ZkTestServer.run start zk server on port:60352
   [junit4]   2> 440998 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 441006 T1320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16b69d91 name:ZooKeeperConnection Watcher:127.0.0.1:60352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 441007 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 441010 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 441015 T1322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c8289d7 name:ZooKeeperConnection Watcher:127.0.0.1:60352 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 441015 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 441024 T1315 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18988c0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 441026 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 441032 T1324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79c280d8 name:ZooKeeperConnection Watcher:127.0.0.1:60352 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 441032 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 441032 T1245 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 441044 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 441048 T1315 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18988c0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 441056 T1245 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 441064 T1245 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 441070 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 441081 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 441086 T1326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1731b721 name:ZooKeeperConnection Watcher:127.0.0.1:60352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 441086 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 441091 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 441101 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 441113 T1320 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 441114 T1326 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 441115 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 441119 T1328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48462da5 name:ZooKeeperConnection Watcher:127.0.0.1:60352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 441120 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 441122 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 441127 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 441138 T1245 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60352_solr
   [junit4]   2> 441138 T1245 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 441144 T1245 oasc.Overseer.start Overseer (id=91669688581095428-127.0.0.1:60352_solr-n_0000000000) starting
   [junit4]   2> 441148 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 441157 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 441165 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 441172 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 441180 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 441188 T1245 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 441201 T1330 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 441204 T1330 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60352_solr
   [junit4]   2> 441206 T1329 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 441210 T1329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 441211 T1329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 441212 T1329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 441212 T1329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 441218 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441220 T1326 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 441220 T1320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 441714 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 441758 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 441783 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441787 T1245 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 441794 T1329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 441796 T1320 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 441796 T1326 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 441799 T1315 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18988c0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 441802 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 441805 T1332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3865c140 name:ZooKeeperConnection Watcher:127.0.0.1:60352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 441805 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 441806 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441808 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 441812 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 441816 T1320 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 441816 T1332 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 441821 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441821 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 441822 T1329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 441822 T1329 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 441826 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441833 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 441841 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441847 T1328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 441952 T1332 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 441953 T1320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 441954 T1245 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 441958 T1315 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18988c0004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 441960 T1332 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 441962 T1315 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18988c0005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 441963 T1320 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 441965 T1245 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60352 60352
   [junit4]   2> 443480 T1329 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:96)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:169)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:493)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:267)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 444251 T1329 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60352_solr
   [junit4]   2> 444255 T1333 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 444258 T1314 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60352 60352
   [junit4]   2> 444264 T1245 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 444278 T1245 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 444282 T1245 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 444284 T1334 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 444385 T1245 oasc.ZkTestServer.run start zk server on port:60373
   [junit4]   2> 444388 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444401 T1340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c1d92d2 name:ZooKeeperConnection Watcher:127.0.0.1:60373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444401 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444404 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444410 T1342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10f47709 name:ZooKeeperConnection Watcher:127.0.0.1:60373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444410 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444416 T1335 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18a5ca0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 444418 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444425 T1344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c00f229 name:ZooKeeperConnection Watcher:127.0.0.1:60373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444425 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444426 T1245 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 444437 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 444451 T1245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444452 T1345 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444456 T1347 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4abd567c name:ZooKeeperConnection Watcher:127.0.0.1:60373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444456 T1245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444459 T1349 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54790d39 name:ZooKeeperConnection Watcher:127.0.0.1:60373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444459 T1345 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444459 T1245 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 444463 T1345 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 444470 T1245 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 444471 T1345 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 444476 T1245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 444491 T1345 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60373_solr
   [junit4]   2> 444492 T1345 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 444492 T1245 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 444503 T1345 oasc.Overseer.start Overseer (id=91669688803262468-127.0.0.1:60373_solr-n_0000000000) starting
   [junit4]   2> 444504 T1347 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 444514 T1245 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 444515 T1345 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 444533 T1345 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 444549 T1345 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 444562 T1345 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 444570 T1345 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 444579 T1345 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 444595 T1345 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 444595 T1351 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 444600 T1335 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18a5ca0004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 444601 T1345 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444603 T1353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e033a3b name:ZooKeeperConnection Watcher:127.0.0.1:60373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444604 T1345 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444605 T1351 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=91669688803262468-127.0.0.1:60373_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 444607 T1351 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60373_solr
   [junit4]   2> 444608 T1351 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:212)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 444615 T1345 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60373_solr
   [junit4]   2> 444615 T1345 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 444620 T1345 oasc.Overseer.start Overseer (id=91669688803262469-127.0.0.1:60373_solr-n_0000000001) starting
   [junit4]   2> 444641 T1355 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 444642 T1355 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:60373_solr
   [junit4]   2> 444643 T1354 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 444647 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 444648 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"s

[...truncated too long message...]

hard1
   [junit4]   2> 474563 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 475476 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 476385 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 477368 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 478374 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 479376 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 480380 T1245 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 480400 T1601 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18f2230003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 480400 T1610 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 480406 T1601 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145ad18f2230005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 480413 T1245 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60637 60637
   [junit4]   2> 481888 T1612 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 482657 T1600 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60637 60637
   [junit4]   2> 482662 T1245 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 482662 T1620 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60637_solr
   [junit4]   2> 482665 T1622 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=43938CF4049E4FF0 -Dtests.slow=true -Dtests.locale=ar_SD -Dtests.timezone=Etc/GMT+3 -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   18.9s | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([43938CF4049E4FF0:479B0307163BA0D1]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
   [junit4]    > 	... 46 more
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
   [junit4]    > 	... 47 more
   [junit4]   2> 482680 T1245 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 482680 T1245 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 485664 T1245 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solr.cloud.OverseerTest-43938CF4049E4FF0-001
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_SD, timezone=Etc/GMT+3
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=2,threads=1,free=176356184,total=270532608
   [junit4]   2> NOTE: All tests run in this JVM: [AnalysisAfterCoreReloadTest, SuggesterFSTTest, TestCSVLoader, TestCharFilters, TestLMJelinekMercerSimilarityFactory, CurrencyFieldOpenExchangeTest, SOLR749Test, TestUniqueKeyFieldResource, TestCSVResponseWriter, TestRemoteStreaming, TestSolrCoreProperties, AnalysisErrorHandlingTest, SpatialFilterTest, TestShardHandlerFactory, TestCopyFieldCollectionResource, TestRangeQuery, TestCloudInspectUtil, TestBinaryField, TestDocBasedVersionConstraints, TestFileDictionaryLookup, TestInfoStreamLogging, TestAnalyzedSuggestions, CoreMergeIndexesAdminHandlerTest, FileBasedSpellCheckerTest, ClusterStateTest, TestUpdate, SpellCheckCollatorTest, TestLazyCores, TestClassNameShortening, TestPseudoReturnFields, ShardRoutingTest, ConnectionManagerTest, PolyFieldTest, TestManagedSynonymFilterFactory, TestNRTOpen, TermVectorComponentDistributedTest, TestIndexSearcher, TestCloudManagedSchemaAddField, LeaderElectionIntegrationTest, MBeansHandlerTest, MoreLikeThisHandlerTest, StandardRequestHandlerTest, TestCollationField, TestCursorMarkWithoutUniqueKey, TestSolr4Spatial, TestConfig, TestSolrDeletionPolicy1, SpellCheckComponentTest, TestStressReorder, OverseerTest]
   [junit4] Completed in 61.75s, 9 tests, 1 error <<< FAILURES!

[...truncated 1059 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:447: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:45: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1297: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:921: There were test failures: 389 suites, 1626 tests, 1 error, 42 ignored (21 assumptions)

Total time: 101 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_55 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message