lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-6.x - Build # 240 - Failure
Date Wed, 01 Jun 2016 00:45:46 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/240/

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior

Error Message:
Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={   "shards":{"shard1":{       "parent":null,       "range":null,       "state":"active",       "replicas":{"core_node1":{           "base_url":"http://127.0.0.1/solr",           "node_name":"node1",           "core":"core1",           "roles":"",           "state":"down"}}}},   "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}

Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
  "shards":{"shard1":{
      "parent":null,
      "range":null,
      "state":"active",
      "replicas":{"core_node1":{
          "base_url":"http://127.0.0.1/solr",
          "node_name":"node1",
          "core":"core1",
          "roles":"",
          "state":"down"}}}},
  "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
	at __randomizedtesting.SeedInfo.seed([3BAE7D3244743A1C:53B07EDEA6E46052]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:243)
	at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1273)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10575 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_3BAE7D3244743A1C-001/init-core-data-001
   [junit4]   2> 0    INFO  (SUITE-OverseerTest-seed#[3BAE7D3244743A1C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 49   INFO  (SUITE-OverseerTest-seed#[3BAE7D3244743A1C]-worker) [    ] o.e.j.u.log Logging initialized @6394ms
   [junit4]   2> 304  INFO  (SUITE-OverseerTest-seed#[3BAE7D3244743A1C]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 304  INFO  (SUITE-OverseerTest-seed#[3BAE7D3244743A1C]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 323  INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalClusterStateChangeBehavior
   [junit4]   2> 331  INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 343  INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 347  INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 543  INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ZkTestServer start zk server on port:50631
   [junit4]   2> 570  INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 995  INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1163 INFO  (zkCallback-1-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2265acde name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1166 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1168 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1170 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1185 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1210 INFO  (zkCallback-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@268d4c82 name:ZooKeeperConnection Watcher:127.0.0.1:50631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1211 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1227 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1279 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1314 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1354 INFO  (zkCallback-3-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@54834da8 name:ZooKeeperConnection Watcher:127.0.0.1:50631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1355 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1355 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1356 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1407 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1419 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1431 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1443 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1459 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1488 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1495 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1514 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1579 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1598 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1618 INFO  (zkCallback-5-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@42b0f35f name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1619 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1619 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2025 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 2072 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2084 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 2089 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2091 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95993145138544643-127.0.0.1:50631_solr-n_0000000000
   [junit4]   2> 2092 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50631_solr
   [junit4]   2> 2094 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 2102 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.Overseer Overseer (id=95993145138544643-127.0.0.1:50631_solr-n_0000000000) starting
   [junit4]   2> 2114 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2131 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2139 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 2142 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 2145 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 2249 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 2269 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2356 INFO  (OverseerCollectionConfigSetProcessor-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 2465 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2466 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2467 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 2469 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 2476 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"down"} current state version: 0
   [junit4]   2> 2481 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"down"}
   [junit4]   2> 2483 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2508 INFO  (zkCallback-1-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2518 INFO  (zkCallback-5-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2525 INFO  (zkCallback-1-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2574 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"} current state version: 1
   [junit4]   2> 2574 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 2574 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
   [junit4]   2> 2574 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 2578 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 2578 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 2579 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
   [junit4]   2> 2579 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 2580 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"test",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "stateFormat":"2"} current state version: 1
   [junit4]   2> 2581 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: test
   [junit4]   2> 2583 WARN  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state
   [junit4]   2> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:365)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:362)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:362)
   [junit4]   2> 	at org.apache.solr.cloud.overseer.ZkStateWriter.writePendingUpdates(ZkStateWriter.java:252)
   [junit4]   2> 	at org.apache.solr.cloud.overseer.ZkStateWriter.enqueueUpdate(ZkStateWriter.java:120)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:217)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2587 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 2588 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"test",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 2588 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: test
   [junit4]   2> 2608 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/test/state.json
   [junit4]   2> 2610 INFO  (zkCallback-5-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2615 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"createshard",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "replicationFactor":"1"} current state version: 2
   [junit4]   2> 2618 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"down"} current state version: 2
   [junit4]   2> 2626 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"down"} 
   [junit4]   2> 2804 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/test/state.json version: 0
   [junit4]   2> 8492 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50631 50631
   [junit4]   2> 8617 INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50631 50631
   [junit4]   2> 8628 WARN  (Thread-14) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> 8628 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalClusterStateChangeBehavior
   [junit4]   2> 8628 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.Overseer Overseer (id=95993145138544643-127.0.0.1:50631_solr-n_0000000000) closing
   [junit4]   2> 8629 INFO  (OverseerStateUpdate-95993145138544643-127.0.0.1:50631_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50631_solr
   [junit4]   2> 8634 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:268)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testExternalClusterStateChangeBehavior -Dtests.seed=3BAE7D3244743A1C -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=he-IL -Dtests.timezone=Europe/Kiev -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 8.33s J2 | OverseerTest.testExternalClusterStateChangeBehavior <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "parent":null,
   [junit4]    >       "range":null,
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "base_url":"http://127.0.0.1/solr",
   [junit4]    >           "node_name":"node1",
   [junit4]    >           "core":"core1",
   [junit4]    >           "roles":"",
   [junit4]    >           "state":"down"}}}},
   [junit4]    >   "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3BAE7D3244743A1C:53B07EDEA6E46052]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:243)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1273)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 8643 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testShardLeaderChange
   [junit4]   2> 8643 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 8644 INFO  (Thread-15) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 8644 INFO  (Thread-15) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 8744 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ZkTestServer start zk server on port:56339
   [junit4]   2> 8744 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 8745 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 8751 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6dfb337a name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8751 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 8752 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 8752 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 8754 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 8759 INFO  (zkCallback-10-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@199db769 name:ZooKeeperConnection Watcher:127.0.0.1:56339 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8759 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 8759 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 8762 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 8762 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 8772 INFO  (zkCallback-11-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@43a7d406 name:ZooKeeperConnection Watcher:127.0.0.1:56339 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8772 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 8772 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 8772 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 8785 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 8787 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 8789 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 8791 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 8793 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 8795 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 8795 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 8796 INFO  (Thread-16) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 8797 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 8799 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 8802 INFO  (zkCallback-13-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@66dad0f1 name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8802 INFO  (Thread-16) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 8811 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 8811 INFO  (Thread-16) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 8814 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 8815 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 8816 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 8833 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 8837 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 8839 INFO  (Thread-16) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95993145674039299-127.0.0.1:56339_solr-n_0000000000
   [junit4]   2> 8840 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@684b9042 name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8839 INFO  (Thread-16) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56339_solr
   [junit4]   2> 8847 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 8848 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=95993145674039299-127.0.0.1:56339_solr-n_0000000000) starting
   [junit4]   2> 8849 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 8849 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 8850 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 8859 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 8861 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 8861 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 8862 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 8863 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 8863 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 8864 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
   [junit4]   2> 8871 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 8871 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 8872 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 8872 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 8875 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 8877 INFO  (Thread-16) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 8890 INFO  (Thread-16) [    ] o.a.s.c.OverseerTest Killing overseer.
   [junit4]   2> 8891 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 8889 INFO  (OverseerCollectionConfigSetProcessor-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 8898 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 8899 INFO  (OverseerStateUpdate-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 8900 INFO  (OverseerStateUpdate-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.Overseer 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:418)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:145)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 8900 INFO  (OverseerStateUpdate-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.Overseer According to ZK I (id=95993145674039299-127.0.0.1:56339_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 8900 INFO  (OverseerStateUpdate-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:56339_solr
   [junit4]   2> 8900 INFO  (Thread-16) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 8904 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:268)
   [junit4]   2> 8907 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c3a8184 name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8907 INFO  (Thread-16) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 8907 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 8907 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=95993145674039299-127.0.0.1:56339_solr-n_0000000000) closing
   [junit4]   2> 8907 INFO  (Thread-16) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 8908 INFO  (OverseerCollectionConfigSetProcessor-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=95993145674039299-127.0.0.1:56339_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 8908 ERROR (OverseerCollectionConfigSetProcessor-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskQueue.getTailId(OverseerTaskQueue.java:282)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:147)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 8908 ERROR (OverseerCollectionConfigSetProcessor-95993145674039299-127.0.0.1:56339_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:60)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:162)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 8911 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 8912 INFO  (Thread-16) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95993145674039301-127.0.0.1:56339_solr-n_0000000001
   [junit4]   2> 8912 INFO  (Thread-16) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56339_solr
   [junit4]   2> 8912 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 8913 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=95993145674039301-127.0.0.1:56339_solr-n_0000000001) starting
   [junit4]   2> 8918 INFO  (Thread-16) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 8923 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 8925 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 8926 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 8926 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 8929 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 0
   [junit4]   2> 8929 INFO  (OverseerCollectionConfigSetProcessor-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 8929 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 8938 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 8941 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 8945 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 8945 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 8945 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 9401 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 9404 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 9404 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 9406 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 9418 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000000
   [junit4]   2> 9425 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 9430 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000000
   [junit4]   2> 9441 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 9443 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000000
   [junit4]   2> 9444 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 1
   [junit4]   2> 9444 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 1
   [junit4]   2> 9445 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 9445 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 9447 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 9459 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000001
   [junit4]   2> 9459 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000001
   [junit4]   2> 9461 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 9462 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 9467 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 9469 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@651c7749 name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 9469 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 9470 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 9471 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 9471 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 9473 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 9474 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
   [junit4]   2> 9475 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 9475 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 9477 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000001
   [junit4]   2> 9478 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 2
   [junit4]   2> 9479 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 9479 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 9479 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 9479 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 9479 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 1
   [junit4]   2> 9479 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 9480 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 9480 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 9482 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000002
   [junit4]   2> 9483 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000002
   [junit4]   2> 9485 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 9488 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 9489 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 9588 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9588 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9588 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9593 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9687 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node2",
   [junit4]   2>   "core":"core4",
   [junit4]   2>   "core_node_name":"node2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node2/solr/"} current state version: 2
   [junit4]   2> 9687 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node2",
   [junit4]   2>   "core":"core4",
   [junit4]   2>   "core_node_name":"node2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node2/solr/"}
   [junit4]   2> 9687 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=1
   [junit4]   2> 9687 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 9791 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9791 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9791 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 9791 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 10187 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 10190 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039302-node2_core4-n_0000000003
   [junit4]   2> 10192 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000002 to know if I could be the leader
   [junit4]   2> 10192 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993145674039300-node1_core1-n_0000000002
   [junit4]   2> 10192 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 3
   [junit4]   2> 10194 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039302-node2_core4-n_0000000003
   [junit4]   2> 10194 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 10194 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 10195 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 10195 INFO  (zkCallback-23-thread-2) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 10198 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 10198 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 10199 INFO  (zkCallback-23-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 10200 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node2/solr/",
   [junit4]   2>   "core":"core4",
   [junit4]   2>   "state":"active"} current state version: 3
   [junit4]   2> 10304 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 10304 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 10304 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11695 WARN  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 11695 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 11696 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 11699 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@240729b7 name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 11700 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 11700 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 11705 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 11705 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 11707 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 11709 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
   [junit4]   2> 11710 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11710 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11710 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11710 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11711 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 11711 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 11712 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 11715 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 11718 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 11719 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 4
   [junit4]   2> 11719 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 11720 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 11729 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000004
   [junit4]   2> 11729 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95993145674039302-node2_core4-n_0000000003 to know if I could be the leader
   [junit4]   2> 11730 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993145674039302-node2_core4-n_0000000003
   [junit4]   2> 11730 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 4
   [junit4]   2> 11731 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11731 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11731 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11732 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 11737 INFO  (Thread-16) [    ] o.a.s.c.OverseerTest Killing overseer.
   [junit4]   2> 11737 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 11737 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 11743 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1550941e4c60006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 11747 INFO  (zkCallback-25-thread-2) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11747 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000004
   [junit4]   2> 11748 INFO  (zkCallback-25-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 11750 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 11750 INFO  (Thread-16) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 11755 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000004
   [junit4]   2> 11755 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 5
   [junit4]   2> 11756 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 11762 INFO  (zkCallback-27-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28afe4b5 name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 11762 INFO  (Thread-16) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 11766 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 11766 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=95993145674039301-127.0.0.1:56339_solr-n_0000000001) closing
   [junit4]   2> 11767 INFO  (Thread-16) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 11767 INFO  (OverseerStateUpdate-95993145674039301-127.0.0.1:56339_solr-n_0000000001) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:56339_solr
   [junit4]   2> 11781 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000005
   [junit4]   2> 11782 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000005
   [junit4]   2> 11783 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 11787 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:268)
   [junit4]   2> 11790 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 11792 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 11792 INFO  (Thread-16) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95993145674039304-127.0.0.1:56339_solr-n_0000000002
   [junit4]   2> 11793 INFO  (Thread-16) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56339_solr
   [junit4]   2> 11793 INFO  (Thread-16) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 11794 INFO  (Thread-16) [    ] o.a.s.c.Overseer Overseer (id=95993145674039304-127.0.0.1:56339_solr-n_0000000002) starting
   [junit4]   2> 11801 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@72caf67a name:ZooKeeperConnection Watcher:127.0.0.1:56339/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 11801 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 11801 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 11804 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 11804 INFO  (Thread-16) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 11805 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 11805 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 11805 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 11806 INFO  (OverseerCollectionConfigSetProcessor-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 11806 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 11807 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
   [junit4]   2> 11807 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 11808 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11808 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 11808 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 11809 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 11809 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 11810 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000005
   [junit4]   2> 11811 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 6
   [junit4]   2> 11812 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 11813 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.Overseer processMessage: workQueueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 11814 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 11814 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 11815 INFO  (zkCallback-25-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11815 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11818 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.Overseer processMessage: queueSize: 4, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 5
   [junit4]   2> 11818 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 11818 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 11819 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000006
   [junit4]   2> 11820 INFO  (TEST-OverseerTest.testShardLeaderChange-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/95993145674039303-node1_core1-n_0000000006
   [junit4]   2> 11820 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.Overseer processMessage: queueSize: 4, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 5
   [junit4]   2> 11823 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.Overseer processMessage: queueSize: 4, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 5
   [junit4]   2> 11823 INFO  (zkCallback-27-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 11825 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.Overseer processMessage: queueSize: 4, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 5
   [junit4]   2> 11825 INFO  (OverseerStateUpdate-95993145674039304-127.0.0.1:56339_solr-n_0000000002) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name"

[...truncated too long message...]

pe:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51001 INFO  (zkCallback-223-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51001 INFO  (zkCallback-218-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51397 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 51403 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95993148266381316-127.0.0.1_core6-n_0000000001
   [junit4]   2> 51403 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95993148266381316-127.0.0.1_core3-n_0000000000 to know if I could be the leader
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993148266381316-127.0.0.1_core6-n_0000000001
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard3/election/95993148266381316-127.0.0.1_core5-n_0000000001
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard2/election/95993148266381316-127.0.0.1_core4-n_0000000001
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 51405 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard1/election/95993148266381316-127.0.0.1_core3-n_0000000000
   [junit4]   2> 51406 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard1/leader 1
   [junit4]   2> 51406 WARN  (zkCallback-223-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 51406 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard3/election/95993148266381316-127.0.0.1_core2-n_0000000000
   [junit4]   2> 51406 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard3/leader 1
   [junit4]   2> 51406 WARN  (zkCallback-223-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 51406 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ElectionContext Canceling election /collections/collection1/leader_elect/shard2/election/95993148266381316-127.0.0.1_core1-n_0000000000
   [junit4]   2> 51407 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ShardLeaderElectionContextBase Removing leader registration node on cancel: /collections/collection1/leaders/shard2/leader 1
   [junit4]   2> 51407 WARN  (zkCallback-223-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 51407 INFO  (zkCallback-223-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51407 INFO  (zkCallback-218-thread-2) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51407 INFO  (zkCallback-223-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 51407 INFO  (zkCallback-218-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 51411 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52311 52311
   [junit4]   2> 51603 INFO  (Thread-26) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52311 52311
   [junit4]   2> 51604 WARN  (Thread-26) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 
   [junit4]   2> 51606 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testShardAssignment
   [junit4]   2> 51606 INFO  (TEST-OverseerTest.testShardAssignment-seed#[3BAE7D3244743A1C]) [    ] o.a.s.c.Overseer Overseer (id=95993148266381315-127.0.0.1:52311_solr-n_0000000000) closing
   [junit4]   2> 51607 INFO  (OverseerStateUpdate-95993148266381315-127.0.0.1:52311_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:52311_solr
   [junit4]   2> 51614 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:268)
   [junit4]   2> 54612 INFO  (SUITE-OverseerTest-seed#[3BAE7D3244743A1C]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_3BAE7D3244743A1C-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=191, maxMBSortInHeap=5.145436812447585, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=he-IL, timezone=Europe/Kiev
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=241029696,total=265814016
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldCacheWithThreads, OverseerTest]
   [junit4] Completed [7/612 (1!)] on J2 in 55.33s, 13 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 1972 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:740: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/common-build.xml:531: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/lucene/common-build.xml:1427: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2564 tests, 1 failure, 87 ignored (72 assumptions) [seed: 3BAE7D3244743A1C]

Total time: 72 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




Mime
View raw message