lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 788 - Failure
Date Thu, 21 Jan 2016 15:36:31 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/788/

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior

Error Message:
Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={   "shards":{"shard1":{       "parent":null,       "range":null,       "state":"active",       "replicas":{"core_node1":{           "base_url":"http://127.0.0.1/solr",           "node_name":"node1",           "core":"core1",           "roles":"",           "state":"down"}}}},   "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}

Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
  "shards":{"shard1":{
      "parent":null,
      "range":null,
      "state":"active",
      "replicas":{"core_node1":{
          "base_url":"http://127.0.0.1/solr",
          "node_name":"node1",
          "core":"core1",
          "roles":"",
          "state":"down"}}}},
  "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
	at __randomizedtesting.SeedInfo.seed([4CEB1583D4F696F8:24F5166F3666CCB6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:237)
	at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1264)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9920 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_4CEB1583D4F696F8-001/init-core-data-001
   [junit4]   2> 397862 INFO  (SUITE-OverseerTest-seed#[4CEB1583D4F696F8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 397862 INFO  (SUITE-OverseerTest-seed#[4CEB1583D4F696F8]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 397862 INFO  (SUITE-OverseerTest-seed#[4CEB1583D4F696F8]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 397870 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPlaceholders
   [junit4]   2> 397870 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 397874 INFO  (Thread-1318) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 397874 INFO  (Thread-1318) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 397974 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer start zk server on port:36923
   [junit4]   2> 397974 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 397974 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 397980 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4510d6cb name:ZooKeeperConnection Watcher:127.0.0.1:36923/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 397980 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 397981 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 397981 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 397981 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 397983 INFO  (zkCallback-291-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f6ad3c8 name:ZooKeeperConnection Watcher:127.0.0.1:36923 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 397983 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 397983 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 397984 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 397985 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 397986 INFO  (zkCallback-292-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@48f5a743 name:ZooKeeperConnection Watcher:127.0.0.1:36923 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 397986 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 397986 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 397986 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 397992 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 397993 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 397994 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 397996 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 397997 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 397998 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 398001 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 398010 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 398018 INFO  (zkCallback-293-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@41adff51 name:ZooKeeperConnection Watcher:127.0.0.1:36923/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 398018 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 398018 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 398020 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 398024 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
   [junit4]   2> 398025 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 398026 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 398027 INFO  (zkCallback-293-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 398042 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 398048 INFO  (zkCallback-294-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@47810996 name:ZooKeeperConnection Watcher:127.0.0.1:36923/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 398048 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 398048 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 398048 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 398056 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 398057 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 398058 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 398059 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95249288280604676-127.0.0.1:36923_solr-n_0000000000
   [junit4]   2> 398060 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36923_solr
   [junit4]   2> 398060 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 398061 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288280604676-127.0.0.1:36923_solr-n_0000000000) starting
   [junit4]   2> 398062 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 398063 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 398064 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 398067 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 398068 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 398069 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 398073 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 398083 INFO  (OverseerCollectionConfigSetProcessor-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 398083 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 398084 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 398087 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"12",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 0
   [junit4]   2> 398088 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=12 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"12",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 398088 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 398089 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 398090 INFO  (zkCallback-294-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 398090 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 398091 INFO  (zkCallback-293-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 398586 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 398587 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 398587 INFO  (zkCallback-294-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 398587 INFO  (zkCallback-293-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 398592 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard2/election/95249288280604675-node1_core1-n_0000000000
   [junit4]   2> 398593 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 398597 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 398600 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 398602 INFO  (zkCallback-293-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 398602 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 400087 INFO  (OverseerCollectionConfigSetProcessor-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 400088 INFO  (OverseerCollectionConfigSetProcessor-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=95249288280604676-127.0.0.1:36923_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 400101 WARN  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.removeFirst(DistributedQueue.java:385)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:188)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:230)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 400101 INFO  (OverseerStateUpdate-95249288280604676-127.0.0.1:36923_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:36923_solr
   [junit4]   2> 400102 WARN  (zkCallback-293-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 400102 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
   [junit4]   2> 400104 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36923 36923
   [junit4]   2> 400314 INFO  (Thread-1318) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36923 36923
   [junit4]   2> 400315 WARN  (Thread-1318) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 
   [junit4]   2> 400315 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPlaceholders
   [junit4]   2> 400315 INFO  (TEST-OverseerTest.testPlaceholders-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288280604676-127.0.0.1:36923_solr-n_0000000000) closing
   [junit4]   2> 400320 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testOverseerFailure
   [junit4]   2> 400320 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 400321 INFO  (Thread-1319) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 400321 INFO  (Thread-1319) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 400421 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer start zk server on port:41638
   [junit4]   2> 400421 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 400422 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 400424 INFO  (zkCallback-297-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1b4e7da9 name:ZooKeeperConnection Watcher:127.0.0.1:41638 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 400424 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 400424 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 400425 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 400426 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 400428 INFO  (zkCallback-298-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e4a1dc name:ZooKeeperConnection Watcher:127.0.0.1:41638 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 400428 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 400428 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 400428 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 400433 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 400433 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 400434 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@790a925b name:ZooKeeperConnection Watcher:127.0.0.1:41638/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 400435 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 400435 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 400436 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 400438 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 400439 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 400440 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 400442 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 400443 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 400445 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 400446 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 400450 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7f26f361 name:ZooKeeperConnection Watcher:127.0.0.1:41638/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 400450 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 400450 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 400453 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 400456 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
   [junit4]   2> 400457 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 400457 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 400457 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 400463 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 400464 INFO  (zkCallback-301-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7560d2ad name:ZooKeeperConnection Watcher:127.0.0.1:41638/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 400464 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 400464 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 400465 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 400466 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 400468 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 400470 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 400470 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95249288440446980-127.0.0.1:41638_solr-n_0000000000
   [junit4]   2> 400471 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41638_solr
   [junit4]   2> 400471 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 400472 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288440446980-127.0.0.1:41638_solr-n_0000000000) starting
   [junit4]   2> 400473 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 400475 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 400477 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 400480 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 400482 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 400483 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 400487 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 400487 INFO  (OverseerCollectionConfigSetProcessor-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 400488 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 400488 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 401491 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 0
   [junit4]   2> 401492 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 401492 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 401492 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 401493 INFO  (zkCallback-301-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 401493 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 401493 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 401989 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 401991 INFO  (zkCallback-301-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 401991 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 401992 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 401997 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000000
   [junit4]   2> 401998 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 402000 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 402003 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://node1/solr/",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 402008 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 402009 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 1
   [junit4]   2> 402009 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 402009 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 402015 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000001
   [junit4]   2> 402015 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000001 , seq(0) /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000000
   [junit4]   2> 402114 INFO  (zkCallback-301-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 402115 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 402115 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 402490 INFO  (OverseerCollectionConfigSetProcessor-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 402490 INFO  (OverseerCollectionConfigSetProcessor-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=95249288440446980-127.0.0.1:41638_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 403169 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 403171 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000002
   [junit4]   2> 403173 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000002 , seq(0) /collections/collection1/leader_elect/shard1/election/95249288440446979-node1_core1-n_0000000000
   [junit4]   2> 403173 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 403174 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 403175 INFO  (zkCallback-304-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@410ead40 name:ZooKeeperConnection Watcher:127.0.0.1:41638/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 403175 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 403175 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 403175 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288440446980-127.0.0.1:41638_solr-n_0000000000) closing
   [junit4]   2> 403175 INFO  (OverseerStateUpdate-95249288440446980-127.0.0.1:41638_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41638_solr
   [junit4]   2> 403176 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 403176 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
   [junit4]   2> 403178 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 403179 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95249288440446981-127.0.0.1:41638_solr-n_0000000001
   [junit4]   2> 403179 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41638_solr
   [junit4]   2> 403179 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 403181 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288440446981-127.0.0.1:41638_solr-n_0000000001) starting
   [junit4]   2> 403188 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 403189 INFO  (OverseerCollectionConfigSetProcessor-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 403189 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 403190 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 403195 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"} current state version: 2
   [junit4]   2> 403196 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"recovering",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://node1/solr/"}
   [junit4]   2> 403196 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 403199 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403199 INFO  (zkCallback-304-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403201 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403252 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"deletecore",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"collection1"} current state version: 3
   [junit4]   2> 403356 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403356 INFO  (zkCallback-304-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403356 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403858 INFO  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403858 INFO  (zkCallback-299-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 403858 INFO  (zkCallback-304-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 405358 WARN  (zkCallback-300-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 405360 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15264b9229a0005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 405366 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41638 41638
   [junit4]   2> 405367 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15264b9229a0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 405575 INFO  (Thread-1319) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41638 41638
   [junit4]   2> 405576 WARN  (Thread-1319) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 405576 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testOverseerFailure
   [junit4]   2> 405576 INFO  (TEST-OverseerTest.testOverseerFailure-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288440446981-127.0.0.1:41638_solr-n_0000000001) closing
   [junit4]   2> 405576 INFO  (OverseerStateUpdate-95249288440446981-127.0.0.1:41638_solr-n_0000000001) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41638_solr
   [junit4]   2> 405590 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
   [junit4]   2> 405592 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalClusterStateChangeBehavior
   [junit4]   2> 405592 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 405594 INFO  (Thread-1320) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 405594 INFO  (Thread-1320) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 405694 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer start zk server on port:60069
   [junit4]   2> 405694 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 405728 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 405747 INFO  (zkCallback-307-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@41667488 name:ZooKeeperConnection Watcher:127.0.0.1:60069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 405747 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 405748 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 405748 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 405761 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 405762 INFO  (zkCallback-308-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3ae9fa77 name:ZooKeeperConnection Watcher:127.0.0.1:60069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 405762 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 405762 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 405764 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 405765 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 405773 INFO  (zkCallback-309-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4bd2521e name:ZooKeeperConnection Watcher:127.0.0.1:60069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 405773 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 405774 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 405774 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 405777 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 405778 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 405780 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 405781 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 405787 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 405788 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 405791 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 405842 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 405850 INFO  (zkCallback-310-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c76a860 name:ZooKeeperConnection Watcher:127.0.0.1:60069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 405850 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 405851 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 405851 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 405853 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 405855 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 405857 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 405858 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95249288786149379-127.0.0.1:60069_solr-n_0000000000
   [junit4]   2> 405859 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60069_solr
   [junit4]   2> 405859 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 405860 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288786149379-127.0.0.1:60069_solr-n_0000000000) starting
   [junit4]   2> 405863 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 405866 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 405880 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 405885 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 405887 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 405889 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 405898 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 405899 INFO  (OverseerCollectionConfigSetProcessor-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 405899 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 405899 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 405914 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"down"} current state version: 0
   [junit4]   2> 405914 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"down"}
   [junit4]   2> 405914 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 405916 INFO  (zkCallback-307-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 405917 INFO  (zkCallback-310-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 405966 INFO  (zkCallback-307-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 405971 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"} current state version: 1
   [junit4]   2> 405972 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 405972 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
   [junit4]   2> 405972 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 405973 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 405974 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"c1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 405974 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
   [junit4]   2> 405974 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
   [junit4]   2> 405976 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"test",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "stateFormat":"2"} current state version: 1
   [junit4]   2> 405976 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: test
   [junit4]   2> 405977 WARN  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state
   [junit4]   2> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:371)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:368)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:368)
   [junit4]   2> 	at org.apache.solr.cloud.overseer.ZkStateWriter.writePendingUpdates(ZkStateWriter.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.overseer.ZkStateWriter.enqueueUpdate(ZkStateWriter.java:120)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:291)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:215)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 405982 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"test",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 405982 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ClusterStateMutator building a new cName: test
   [junit4]   2> 405985 INFO  (zkCallback-310-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 405989 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/test/state.json
   [junit4]   2> 405993 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"createshard",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "replicationFactor":"1"} current state version: 2
   [junit4]   2> 405995 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"down"} current state version: 2
   [junit4]   2> 405995 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "state":"down"} 
   [junit4]   2> 406097 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/test/state.json version: 0
   [junit4]   2> 411143 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60069 60069
   [junit4]   2> 411362 INFO  (Thread-1320) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60069 60069
   [junit4]   2> 411363 WARN  (Thread-1320) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> 411364 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalClusterStateChangeBehavior
   [junit4]   2> 411364 INFO  (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249288786149379-127.0.0.1:60069_solr-n_0000000000) closing
   [junit4]   2> 411364 INFO  (OverseerStateUpdate-95249288786149379-127.0.0.1:60069_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60069_solr
   [junit4]   2> 411364 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testExternalClusterStateChangeBehavior -Dtests.seed=4CEB1583D4F696F8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sq -Dtests.timezone=Asia/Samarkand -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 5.79s J2 | OverseerTest.testExternalClusterStateChangeBehavior <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "parent":null,
   [junit4]    >       "range":null,
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "base_url":"http://127.0.0.1/solr",
   [junit4]    >           "node_name":"node1",
   [junit4]    >           "core":"core1",
   [junit4]    >           "roles":"",
   [junit4]    >           "state":"down"}}}},
   [junit4]    >   "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4CEB1583D4F696F8:24F5166F3666CCB6]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:237)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1264)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 411381 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateChange
   [junit4]   2> 411381 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 411382 INFO  (Thread-1321) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 411382 INFO  (Thread-1321) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 411482 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer start zk server on port:50381
   [junit4]   2> 411482 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 411483 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 411485 INFO  (zkCallback-313-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@47aae24d name:ZooKeeperConnection Watcher:127.0.0.1:50381/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 411485 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 411485 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 411485 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 411486 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 411487 INFO  (zkCallback-314-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2ac5da8e name:ZooKeeperConnection Watcher:127.0.0.1:50381 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 411487 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 411487 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 411488 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 411489 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 411491 INFO  (zkCallback-315-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1dbbbc12 name:ZooKeeperConnection Watcher:127.0.0.1:50381 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 411494 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 411494 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 411494 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 411497 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 411500 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 411501 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 411503 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 411505 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 411506 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 411509 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 411511 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 411512 INFO  (zkCallback-316-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1b15b219 name:ZooKeeperConnection Watcher:127.0.0.1:50381/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 411513 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 411513 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 411513 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 411514 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 411517 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 411518 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 411519 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95249289165275139-127.0.0.1:50381_solr-n_0000000000
   [junit4]   2> 411519 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50381_solr
   [junit4]   2> 411519 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 411521 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249289165275139-127.0.0.1:50381_solr-n_0000000000) starting
   [junit4]   2> 411523 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 411525 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 411527 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 411531 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 411533 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 411536 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 411542 INFO  (TEST-OverseerTest.testStateChange-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 411542 INFO  (OverseerStateUpdate-95249289165275139-127.0.0.1:50381_solr-n_0000000000) [    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 411543 INFO  (OverseerStateUpdate-95249289165275139-127.0.0.1:50381_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 411550 INFO  (OverseerCollectionConfigSetProcessor-95249289165275139-127.0.0.1:50381_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 411562 INFO  (OverseerStateUpdate-95249289165275139-127.0.0.1:50381_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"} current state version: 0
   [junit4]   2> 411563 INFO  (OverseerStateUpdate-95249289165275139-127.0.0.1:50381_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 411563 INFO  (OverseerStateUpdate-95249289165275139-127.0.0.1:50381_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shar

[...truncated too long message...]

008_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 452315 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "shard":"s1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 452316 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "shard":"s1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core1",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 452317 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "shard":"s1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core2",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 452319 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "shard":"s1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core2",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 452319 INFO  (zkCallback-417-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 452319 INFO  (zkCallback-420-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 452320 INFO  (zkCallback-417-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 452321 INFO  (OverseerCollectionConfigSetProcessor-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 452322 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "shard":"s1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core3",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"} current state version: 2
   [junit4]   2> 452322 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1/solr",
   [junit4]   2>   "node_name":"node1",
   [junit4]   2>   "shard":"s1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"core3",
   [junit4]   2>   "roles":"",
   [junit4]   2>   "state":"recovering"}
   [junit4]   2> 452326 INFO  (zkCallback-420-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 452425 INFO  (zkCallback-417-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 452425 INFO  (zkCallback-420-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 452521 INFO  (TEST-OverseerTest.testReplay-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37008 37008
   [junit4]   2> 452698 INFO  (Thread-1330) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37008 37008
   [junit4]   2> 452701 WARN  (Thread-1330) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> 452701 INFO  (TEST-OverseerTest.testReplay-seed#[4CEB1583D4F696F8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testReplay
   [junit4]   2> 452701 INFO  (TEST-OverseerTest.testReplay-seed#[4CEB1583D4F696F8]) [    ] o.a.s.c.Overseer Overseer (id=95249291838750723-127.0.0.1:37008_solr-n_0000000000) closing
   [junit4]   2> 452701 INFO  (OverseerStateUpdate-95249291838750723-127.0.0.1:37008_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37008_solr
   [junit4]   2> 452702 ERROR (OverseerExitThread) [    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
   [junit4]   2> 455703 INFO  (SUITE-OverseerTest-seed#[4CEB1583D4F696F8]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_4CEB1583D4F696F8-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=sq, timezone=Asia/Samarkand
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=134353336,total=463470592
   [junit4]   2> NOTE: All tests run in this JVM: [TestUtils, UnloadDistributedZkTest, SpatialHeatmapFacetsTest, SolrCoreCheckLockOnStartupTest, TestCollationField, LeaderElectionTest, PrimitiveFieldTypeTest, CurrencyFieldXmlFileTest, TestSuggestSpellingConverter, TestCoreDiscovery, TestCollationFieldDocValues, TestSolr4Spatial2, CircularListTest, SimpleFacetsTest, DebugComponentTest, DataDrivenBlockJoinTest, TestIndexingPerformance, RankQueryTest, TestRestoreCore, LeaderFailoverAfterPartitionTest, TestExactSharedStatsCache, StressHdfsTest, BigEndianAscendingWordDeserializerTest, StatelessScriptUpdateProcessorFactoryTest, HdfsCollectionsAPIDistributedZkTest, OverseerTest]
   [junit4] Completed [101/577 (1!)] on J2 in 57.89s, 13 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 1538 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:784: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1424: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:981: There were test failures: 577 suites (8 ignored), 2316 tests, 1 failure, 85 ignored (70 assumptions) [seed: 4CEB1583D4F696F8]

Total time: 59 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message