lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Windows (32bit/jdk1.8.0_05) - Build # 3919 - Failure!
Date Thu, 01 May 2014 13:06:10 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3919/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseConcMarkSweepGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
commitWithin did not work expected:<68> but was:<67>

Stack Trace:
java.lang.AssertionError: commitWithin did not work expected:<68> but was:<67>
	at __randomizedtesting.SeedInfo.seed([4F0E5599A66BEBC2:CEE8DB81D1348BFE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:357)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
	at __randomizedtesting.SeedInfo.seed([4F0E5599A66BEBC2:4B06DA6AB4CE04E3]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11184 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solr.cloud.OverseerTest-4F0E5599A66BEBC2-001\init-core-data-001
   [junit4]   2> 1081338 T4014 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1081338 T4014 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1081338 T4014 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1081342 T4014 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 1081345 T4014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1081346 T4015 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1081462 T4014 oasc.ZkTestServer.run start zk server on port:64919
   [junit4]   2> 1081464 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081481 T4021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@189c19 name:ZooKeeperConnection Watcher:127.0.0.1:64919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081483 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1081484 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081487 T4023 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@120713c name:ZooKeeperConnection Watcher:127.0.0.1:64919 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081487 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1081491 T4016 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be63810001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1081493 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081496 T4025 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cb65a2 name:ZooKeeperConnection Watcher:127.0.0.1:64919 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081496 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1081496 T4014 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1081503 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1081503 T4016 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be63810002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1081508 T4014 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1081511 T4014 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1081515 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1081521 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1081530 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1081533 T4027 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52803 name:ZooKeeperConnection Watcher:127.0.0.1:64919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1081533 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1081534 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1081537 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1081545 T4014 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64919_solr
   [junit4]   2> 1081545 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1081550 T4014 oasc.Overseer.start Overseer (id=91681395772030979-127.0.0.1:64919_solr-n_0000000000) starting
   [junit4]   2> 1081553 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1081561 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1081568 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1081574 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1081581 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1081595 T4029 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1081597 T4029 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:64919_solr
   [junit4]   2> 1081601 T4028 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1081602 T4028 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 1081603 T4028 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1081605 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1081611 T4028 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1081614 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1081617 T4028 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1081621 T4028 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1081622 T4028 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1081626 T4027 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1081629 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1081723 T4016 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be63810000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1081724 T4014 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64919 64919
   [junit4]   2> 1084204 T4015 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64919 64919
   [junit4]   2> 1084206 T4014 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 1084207 T4028 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:64919_solr
   [junit4]   2> 1084208 T4030 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 1084213 T4014 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 1084216 T4014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1084217 T4031 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1084322 T4014 oasc.ZkTestServer.run start zk server on port:64934
   [junit4]   2> 1084324 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1084330 T4037 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19954e name:ZooKeeperConnection Watcher:127.0.0.1:64934/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1084330 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1084332 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1084334 T4039 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38a060 name:ZooKeeperConnection Watcher:127.0.0.1:64934 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1084335 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1084338 T4032 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be6eb70001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1084339 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1084341 T4041 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22f30 name:ZooKeeperConnection Watcher:127.0.0.1:64934 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1084342 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1084342 T4014 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1084347 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1084347 T4032 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be6eb70002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1084352 T4014 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1084356 T4014 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1084359 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1084365 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1084367 T4043 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e65d03 name:ZooKeeperConnection Watcher:127.0.0.1:64934/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1084368 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1084369 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1084373 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1084377 T4043 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1084378 T4037 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1084379 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1084381 T4045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f51894 name:ZooKeeperConnection Watcher:127.0.0.1:64934/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1084381 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1084382 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1084386 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1084393 T4014 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64934_solr
   [junit4]   2> 1084394 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1084398 T4014 oasc.Overseer.start Overseer (id=91681395960119300-127.0.0.1:64934_solr-n_0000000000) starting
   [junit4]   2> 1084401 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1084406 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1084411 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1084416 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1084421 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1084427 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1084439 T4047 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1084440 T4047 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:64934_solr
   [junit4]   2> 1084440 T4046 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1084444 T4046 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1084445 T4046 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1084445 T4046 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 1084445 T4046 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1084450 T4045 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1084451 T4043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1084452 T4037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1084967 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1084985 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1084995 T4045 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1084996 T4014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1084997 T4046 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1084998 T4032 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be6eb70004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1085001 T4043 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1085001 T4037 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1085002 T4032 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be6eb70003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1085004 T4032 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be6eb70000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1085005 T4014 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64934 64934
   [junit4]   2> 1086573 T4046 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:239)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:422)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1087373 T4031 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64934 64934
   [junit4]   2> 1087373 T4046 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:64934_solr
   [junit4]   2> 1087375 T4048 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 1087377 T4014 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   2> 1087384 T4014 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 1087386 T4014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1087387 T4049 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1087492 T4014 oasc.ZkTestServer.run start zk server on port:64952
   [junit4]   2> 1087495 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087502 T4055 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ef846a name:ZooKeeperConnection Watcher:127.0.0.1:64952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087503 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1087506 T4050 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be7b190000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1087507 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087509 T4057 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd53e8 name:ZooKeeperConnection Watcher:127.0.0.1:64952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087509 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1087510 T4014 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1087515 T4050 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be7b190001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1087516 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087519 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@181e99 name:ZooKeeperConnection Watcher:127.0.0.1:64952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087519 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1087519 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1087523 T4014 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1087527 T4014 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1087530 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1087535 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087537 T4061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b884f6 name:ZooKeeperConnection Watcher:127.0.0.1:64952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087538 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1087539 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1087543 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1087547 T4059 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1087548 T4061 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1087549 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1087551 T4063 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1109d3f name:ZooKeeperConnection Watcher:127.0.0.1:64952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1087552 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1087553 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1087556 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1087562 T4014 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64952_solr
   [junit4]   2> 1087563 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1087567 T4014 oasc.Overseer.start Overseer (id=91681396167868420-127.0.0.1:64952_solr-n_0000000000) starting
   [junit4]   2> 1087571 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1087576 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1087580 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1087586 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1087591 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1087596 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1087607 T4065 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1087607 T4065 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:64952_solr
   [junit4]   2> 1087608 T4064 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1088660 T4063 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1088664 T4064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1088665 T4064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1088665 T4064 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1088665 T4064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1088671 T4063 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1088672 T4059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1088673 T4061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1089186 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1089203 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1089214 T4063 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1089215 T4014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1089216 T4064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1089219 T4063 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1089220 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1089223 T4064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1089223 T4064 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1089228 T4063 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1089337 T4059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1089337 T4061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1089340 T4050 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be7b190004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1090395 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1090407 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1090409 T4067 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e9a126 name:ZooKeeperConnection Watcher:127.0.0.1:64952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1090410 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1090410 T4064 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:64952_solr
   [junit4]   2> 1090413 T4068 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 1090419 T4014 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64952_solr
   [junit4]   2> 1090419 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1090422 T4014 oasc.Overseer.start Overseer (id=91681396167868421-127.0.0.1:64952_solr-n_0000000001) starting
   [junit4]   2> 1090443 T4070 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1090444 T4070 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:64952_solr
   [junit4]   2> 1090447 T4069 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1090447 T4069 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 1090449 T4059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1090450 T4061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1090454 T4069 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1090454 T4069 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1090456 T4059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1090456 T4061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1090459 T4069 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1090462 T4069 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1090464 T4069 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1090464 T4069 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1090469 T4067 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1090470 T4059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1090471 T4061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1090508 T4067 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1090509 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1090510 T4069 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1090520 T4014 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91681396167868419-node1_core1-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:127)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:483)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1090524 T4014 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91681396167868419-node1_core1-n_0000000000
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:127)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:483)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1090525 T4061 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1090532 T4059 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1090533 T4050 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be7b190003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1090535 T4050 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be7b190005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1090537 T4050 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be7b190002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1090538 T4014 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64952 64952
   [junit4]   2> 1092109 T4061 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders
   [junit4]   2> 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
   [junit4]   2> 	... 6 more
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
   [junit4]   2> 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
   [junit4]   2> 	... 7 more
   [junit4]   2> 
   [junit4]   2> 1092905 T4049 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64952 64952
   [junit4]   2> 1092905 T4061 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1092111 T4069 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:96)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:169)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:493)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:256)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1092910 T4069 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:64952_solr
   [junit4]   2> 1092909 T4014 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 1092909 T4061 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1092911 T4071 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=4F0E5599A66BEBC2 -Dtests.slow=true -Dtests.locale=zh_TW -Dtests.timezone=America/Denver -Dtests.file.encoding=UTF-8
   [junit4] ERROR   5.55s | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4F0E5599A66BEBC2:4B06DA6AB4CE04E3]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1092928 T4014 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 1092930 T4014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1092933 T4072 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1093037 T4014 oasc.ZkTestServer.run start zk server on port:64973
   [junit4]   2> 1093040 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1093046 T4078 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@586cde name:ZooKeeperConnection Watcher:127.0.0.1:64973/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1093046 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1093049 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1093051 T4080 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1787ae1 name:ZooKeeperConnection Watcher:127.0.0.1:64973 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1093051 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1093054 T4073 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be90c20001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1093055 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1093058 T4082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@159bf69 name:ZooKeeperConnection Watcher:127.0.0.1:64973 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1093058 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1093058 T4014 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1093064 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1093064 T4073 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be90c20002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1093068 T4014 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1093071 T4014 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1093074 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1093079 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1093082 T4084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18c9233 name:ZooKeeperConnection Watcher:127.0.0.1:64973/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1093082 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1093084 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1093089 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1093093 T4078 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1093093 T4084 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1093095 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1093097 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d73740 name:ZooKeeperConnection Watcher:127.0.0.1:64973/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1093098 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1093099 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1093102 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1093109 T4014 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64973_solr
   [junit4]   2> 1093109 T4014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1093112 T4014 oasc.Overseer.start Overseer (id=91681396531265540-127.0.0.1:64973_solr-n_0000000000) starting
   [junit4]   2> 1093116 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1093122 T4014 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1093128 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1093134 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1093145 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1093151 T4014 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1093162 T4088 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1093164 T4088 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:64973_solr
   [junit4]   2> 1093167 T4087 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1093169 T4087 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1093170 T4087 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1093170 T4087 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1093170 T4087 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1093175 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093177 T4078 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1093177 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1093656 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1093673 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1093682 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093683 T4014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1093685 T4087 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1093686 T4084 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1093685 T4078 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1093689 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1093691 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093693 T4090 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18e149 name:ZooKeeperConnection Watcher:127.0.0.1:64973/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1093693 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1093695 T4014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1093699 T4014 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1093702 T4078 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1093702 T4090 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1093706 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093706 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1093708 T4087 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1093709 T4087 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1093713 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093715 T4014 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1093723 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093729 T4086 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1093827 T4078 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1093827 T4090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1093829 T4014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1093833 T4078 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1093833 T4090 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1093836 T4073 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be90c20000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1093837 T4014 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64973 64973
   [junit4]   2> 1095249 T4087 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:96)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:169)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:493)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:267)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1095968 T4087 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:64973_solr
   [junit4]   2> 1095968 T4072 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:64973 64973
   [junit4]   2> 1095971 T4091 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 1095971 T4014 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 1095977 T4014 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 1095979 T4014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1095980 T4092 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1096074 T4014 oasc.ZkTestServer.run start zk server on port:64994
   [junit4]   2> 1096077 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1096086 T4098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78b354 name:ZooKeeperConnection Watcher:127.0.0.1:64994 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1096086 T4014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1096089 T4093 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145b7be9ca90000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1096090 T4014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1096092 T4100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@138bba2 name:ZooKeeperConnection Watcher:127.0.0.1:64994 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1096093 T4014 oascc.ConnectionManager.waitForC

[...truncated too long message...]

lr.cloud.BasicDistributedZkTest-4F0E5599A66BEBC2-001\tempDir-001/jetty4\;done=false>>]
   [junit4]   2> 1390179 T5344 oasc.CachingDirectoryFactory.close Closing directory: .\solr.cloud.BasicDistributedZkTest-4F0E5599A66BEBC2-001\tempDir-001/jetty4\
   [junit4]   2> 1390179 T5344 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\solr.cloud.BasicDistributedZkTest-4F0E5599A66BEBC2-001\tempDir-001/jetty4\index [CachedDir<<refCount=0;path=.\solr.cloud.BasicDistributedZkTest-4F0E5599A66BEBC2-001\tempDir-001/jetty4\index;done=false>>]
   [junit4]   2> 1390179 T5344 oasc.CachingDirectoryFactory.close Closing directory: .\solr.cloud.BasicDistributedZkTest-4F0E5599A66BEBC2-001\tempDir-001/jetty4\index
   [junit4]   2> 1390179 T5344 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1390180 T5344 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1390226 T5367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f4 name:ZooKeeperConnection Watcher:127.0.0.1:49299/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 1390226 T5367 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=4F0E5599A66BEBC2 -Dtests.slow=true -Dtests.locale=ar_AE -Dtests.timezone=Antarctica/Rothera -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 24.8s | BasicDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: commitWithin did not work expected:<68> but was:<67>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4F0E5599A66BEBC2:CEE8DB81D1348BFE]:0)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:357)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1390232 T5244 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solr.cloud.BasicDistributedZkTest-4F0E5599A66BEBC2-001
   [junit4]   2> 24813 T5243 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ar_AE, timezone=Antarctica/Rothera
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_05 (32-bit)/cpus=2,threads=1,free=38411584,total=153726976
   [junit4]   2> NOTE: All tests run in this JVM: [DocumentAnalysisRequestHandlerTest, AddSchemaFieldsUpdateProcessorFactoryTest, SortByFunctionTest, ExternalFileFieldSortTest, TestStressUserVersions, TestPhraseSuggestions, TermVectorComponentTest, TestAddFieldRealTimeGet, TestFoldingMultitermQuery, TestFreeTextSuggestions, CollectionsAPIDistributedZkTest, RequestHandlersTest, TestCSVLoader, TestStressVersions, TestWordDelimiterFilterFactory, TestCloudManagedSchemaAddField, TestFastOutputStream, SpatialFilterTest, TestSolrQueryParserResource, TestBinaryField, RegexBoostProcessorTest, PrimUtilsTest, DistributedSuggestComponentTest, TestSolrIndexConfig, TestDistributedGrouping, TestRTGBase, TestCharFilters, QueryResultKeyTest, TestQuerySenderListener, TestElisionMultitermQuery, SpellCheckCollatorTest, TestNumberUtils, TestPseudoReturnFields, TestManagedSchema, TestSchemaVersionResource, TestNoOpRegenerator, TestLMJelinekMercerSimilarityFactory, TimeZoneUtilsTest, TestConfigSets, TestIndexSearcher, BadIndexSchemaTest, PrimitiveFieldTypeTest, SliceStateUpdateTest, TestCloudInspectUtil, SimplePostToolTest, SolrCoreTest, TestCollationKeyRangeQueries, SolrRequestParserTest, HdfsBasicDistributedZkTest, OverseerStatusTest, ChaosMonkeySafeLeaderTest, MoreLikeThisHandlerTest, DistributedQueryComponentOptimizationTest, LeaderElectionIntegrationTest, MigrateRouteKeyTest, TestSchemaNameResource, TestCopyFieldCollectionResource, TestExtendedDismaxParser, TestJmxIntegration, ZkNodePropsTest, BinaryUpdateRequestHandlerTest, RAMDirectoryFactoryTest, DistributedSpellCheckComponentTest, DocExpirationUpdateProcessorFactoryTest, QueryEqualityTest, FileBasedSpellCheckerTest, TestSolrDeletionPolicy2, QueryElevationComponentTest, LegacyHTMLStripCharFilterTest, TestReloadAndDeleteDocs, SolrTestCaseJ4Test, TestSolrCoreProperties, TestSolrQueryParserDefaultOperatorResource, TestFaceting, ResourceLoaderTest, SampleTest, ClusterStateUpdateTest, TestSurroundQueryParser, TestUtils, OutputWriterTest, TestNRTOpen, TestClassNameShortening, UniqFieldsUpdateProcessorFactoryTest, JSONWriterTest, EnumFieldTest, StressHdfsTest, TestArbitraryIndexDir, DateFieldTest, TestIndexingPerformance, DocumentBuilderTest, TestHighlightDedupGrouping, TestReload, TestSolrXml, TestWriterPerf, TestManagedSchemaFieldResource, SuggestComponentTest, FieldMutatingUpdateProcessorTest, ConnectionManagerTest, SuggesterTest, TestManagedResource, CoreAdminRequestStatusTest, PingRequestHandlerTest, TestStressLucene, TestSearchPerf, PathHierarchyTokenizerFactoryTest, TestFiltering, TestBadConfig, TestShortCircuitedRequests, AnalysisAfterCoreReloadTest, LeaderElectionTest, DistributedDebugComponentTest, DeleteInactiveReplicaTest, TestSolrDeletionPolicy1, TestQuerySenderNoQuery, AddBlockUpdateTest, StandardRequestHandlerTest, TestFieldCollectionResource, TestValueSourceCache, TestRealTimeGet, CollectionsAPIAsyncDistributedZkTest, TestLFUCache, SolrPluginUtilsTest, TestRangeQuery, BadComponentTest, FileUtilsTest, DistanceFunctionTest, TestUniqueKeyFieldResource, HdfsLockFactoryTest, InfoHandlerTest, XmlUpdateRequestHandlerTest, TestStandardQParsers, HardAutoCommitTest, OverseerTest, HdfsSyncSliceTest, URLClassifyProcessorTest, TestCoreDiscovery, TestCustomSort, SystemInfoHandlerTest, SliceStateTest, TestPHPSerializedResponseWriter, TestDocBasedVersionConstraints, DOMUtilTest, ShardSplitTest, TestMaxScoreQueryParser, DebugComponentTest, ReturnFieldsTest, TestCollationField, TestRandomDVFaceting, DisMaxRequestHandlerTest, TestCSVResponseWriter, TestSort, StatsComponentTest, TestExceedMaxTermLength, CoreAdminHandlerTest, HdfsUnloadDistributedZkTest, TestFieldResource, DistributedQueryComponentCustomSortTest, TestRecoveryHdfs, TestSimpleQParserPlugin, TestInfoStreamLogging, PeerSyncTest, TestCursorMarkWithoutUniqueKey, DefaultValueUpdateProcessorTest, HdfsBasicDistributedZk2Test, TestIBSimilarityFactory, TestDocSet, ChangedSchemaMergeTest, ZkSolrClientTest, StatelessScriptUpdateProcessorFactoryTest, BasicFunctionalityTest, TestConfig, HdfsRecoveryZkTest, TestGroupingSearch, TestPerFieldSimilarity, HighlighterTest, SuggesterWFSTTest, TestSchemaResource, AsyncMigrateRouteKeyTest, TestFieldTypeCollectionResource, OverseerCollectionProcessorTest, TestCloudManagedSchema, SpellPossibilityIteratorTest, TestJoin, TestFunctionQuery, TestRemoteStreaming, BasicDistributedZkTest]
   [junit4] Completed in 25.26s, 1 test, 1 failure <<< FAILURES!

[...truncated 618 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:447: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:45: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1297: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:921: There were test failures: 389 suites, 1626 tests, 1 error, 1 failure, 42 ignored (21 assumptions)

Total time: 91 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message