lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Woodward <a...@flax.co.uk>
Subject Re: [JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 130 - Still Failing!
Date Sat, 14 May 2016 09:23:27 GMT
OK, so this is a bug in how I'm checking for fully-active collections.  Currently it iterates over all replicas in the collection and checks that they're active, but it isn't checking that all replicas are actually present.  So in this test, the first 'active' check is passing before all shards have been constructed, and then the watch is triggered by the final shards registering themselves, before a node is taken down.

Working on a fix now.

Alan Woodward
www.flax.co.uk


On 14 May 2016, at 09:00, Alan Woodward wrote:

> I'll dig.
> 
> Alan Woodward
> www.flax.co.uk
> 
> 
> On 14 May 2016, at 03:06, Policeman Jenkins Server wrote:
> 
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/130/
>> Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC
>> 
>> 1 tests failed.
>> FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testSimpleCollectionWatch
>> 
>> Error Message:
>> Expected: is <3>      got: <4> 
>> 
>> Stack Trace:
>> java.lang.AssertionError: 
>> Expected: is <3>
>>     got: <4>
>> 
>> 	at __randomizedtesting.SeedInfo.seed([5F36DD2E780EA866:20D125E3F033758]:0)
>> 	at org.junit.Assert.assertThat(Assert.java:780)
>> 	at org.junit.Assert.assertThat(Assert.java:738)
>> 	at org.apache.solr.common.cloud.TestCollectionStateWatchers.testSimpleCollectionWatch(TestCollectionStateWatchers.java:116)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:498)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>> 	at java.lang.Thread.run(Thread.java:745)
>> 
>> 
>> 
>> 
>> Build Log:
>> [...truncated 12855 lines...]
>>   [junit4] Suite: org.apache.solr.common.cloud.TestCollectionStateWatchers
>>   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/init-core-data-001
>>   [junit4]   2> 15517 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
>>   [junit4]   2> 15537 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>>   [junit4]   2> 15539 INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
>>   [junit4]   2> 15540 INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer Starting server
>>   [junit4]   2> 15638 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:49873
>>   [junit4]   2> 15669 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 15726 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 15741 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>   [junit4]   2> 17661 INFO  (zkCallback-7-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c56c092 name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17662 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17662 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17665 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
>>   [junit4]   2> 17696 INFO  (jetty-launcher-6-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>>   [junit4]   2> 17698 INFO  (jetty-launcher-6-thread-4) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>>   [junit4]   2> 17701 INFO  (jetty-launcher-6-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14584893{/solr,null,AVAILABLE}
>>   [junit4]   2> 17705 INFO  (jetty-launcher-6-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>>   [junit4]   2> 17707 INFO  (jetty-launcher-6-thread-3) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>>   [junit4]   2> 17707 INFO  (jetty-launcher-6-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1bc94856{/solr,null,AVAILABLE}
>>   [junit4]   2> 17705 INFO  (jetty-launcher-6-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@3481c7f0{HTTP/1.1,[http/1.1]}{127.0.0.1:36992}
>>   [junit4]   2> 17710 INFO  (jetty-launcher-6-thread-4) [    ] o.e.j.s.Server Started @22289ms
>>   [junit4]   2> 17708 INFO  (jetty-launcher-6-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@502c5f47{HTTP/1.1,[http/1.1]}{127.0.0.1:57008}
>>   [junit4]   2> 17710 INFO  (jetty-launcher-6-thread-1) [    ] o.e.j.s.Server Started @22289ms
>>   [junit4]   2> 17714 INFO  (jetty-launcher-6-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5c08cb7e{/solr,null,AVAILABLE}
>>   [junit4]   2> 17710 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36992}
>>   [junit4]   2> 17714 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57008}
>>   [junit4]   2> 17726 INFO  (jetty-launcher-6-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@5040a9d1{HTTP/1.1,[http/1.1]}{127.0.0.1:63532}
>>   [junit4]   2> 17727 INFO  (jetty-launcher-6-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@34f50864{/solr,null,AVAILABLE}
>>   [junit4]   2> 17728 INFO  (jetty-launcher-6-thread-2) [    ] o.e.j.s.Server Started @22304ms
>>   [junit4]   2> 17728 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=63532}
>>   [junit4]   2> 17728 INFO  (jetty-launcher-6-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@1962ea4f{HTTP/1.1,[http/1.1]}{127.0.0.1:40413}
>>   [junit4]   2> 17728 INFO  (jetty-launcher-6-thread-3) [    ] o.e.j.s.Server Started @22306ms
>>   [junit4]   2> 17725 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
>>   [junit4]   2> 17729 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40413}
>>   [junit4]   2> 17728 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
>>   [junit4]   2> 17728 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
>>   [junit4]   2> 17730 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2'
>>   [junit4]   2> 17729 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
>>   [junit4]   2> 17729 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node1'
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4'
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node3'
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>>   [junit4]   2> 17730 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>   [junit4]   2> 17733 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>>   [junit4]   2> 17733 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17733 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17733 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17731 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17734 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17735 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17737 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17741 INFO  (zkCallback-8-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6f24369d name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17742 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17742 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17742 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@e155deb name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17742 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17743 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17745 INFO  (zkCallback-11-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3a75daa3 name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17745 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 17745 INFO  (zkCallback-10-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2fe67e39 name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17746 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 17746 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17746 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17746 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17747 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17750 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 17750 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 17791 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node1/.
>>   [junit4]   2> 17791 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 1416086828
>>   [junit4]   2> 17792 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node1]
>>   [junit4]   2> 17792 WARN  (jetty-launcher-6-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node1/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node1/lib
>>   [junit4]   2> 17792 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>>   [junit4]   2> 17794 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/.
>>   [junit4]   2> 17794 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 1263981755
>>   [junit4]   2> 17794 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2]
>>   [junit4]   2> 17795 WARN  (jetty-launcher-6-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/lib
>>   [junit4]   2> 17795 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>>   [junit4]   2> 17794 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/.
>>   [junit4]   2> 17799 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.CoreContainer New CoreContainer 527649829
>>   [junit4]   2> 17799 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4]
>>   [junit4]   2> 17799 WARN  (jetty-launcher-6-thread-3) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/lib
>>   [junit4]   2> 17799 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>>   [junit4]   2> 17851 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>>   [junit4]   2> 17851 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>>   [junit4]   2> 17853 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>   [junit4]   2> 17853 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 17853 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>>   [junit4]   2> 17853 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>   [junit4]   2> 17855 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 17855 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49873/solr
>>   [junit4]   2> 17855 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49873/solr
>>   [junit4]   2> 17858 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>   [junit4]   2> 17858 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 17858 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49873/solr
>>   [junit4]   2> 17875 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node3/.
>>   [junit4]   2> 17877 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.CoreContainer New CoreContainer 860529212
>>   [junit4]   2> 17877 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node3]
>>   [junit4]   2> 17877 WARN  (jetty-launcher-6-thread-4) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node3/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node3/lib
>>   [junit4]   2> 17878 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>>   [junit4]   2> 17878 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>>   [junit4]   2> 17879 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>   [junit4]   2> 17879 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 17879 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49873/solr
>>   [junit4]   2> 17889 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2> 17890 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2> 17890 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17890 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17889 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2> 17889 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2> 17890 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17890 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17891 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17892 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17892 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 17894 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17897 INFO  (zkCallback-21-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@60630f0c name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17898 INFO  (zkCallback-22-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1b330ba3 name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17898 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17898 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17898 INFO  (jetty-launcher-6-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17902 INFO  (jetty-launcher-6-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17903 INFO  (zkCallback-23-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@631b7270 name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17904 INFO  (zkCallback-20-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@396c62fb name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17904 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17905 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17907 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17905 INFO  (jetty-launcher-6-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 17910 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$46/1386184206@1347fea0
>>   [junit4]   2> 17910 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$46/1386184206@10d72ed5
>>   [junit4]   2> 17913 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$46/1386184206@3cff41b
>>   [junit4]   2> 17914 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17914 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17915 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17916 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$46/1386184206@2c9cca7a
>>   [junit4]   2> 17917 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@23dc83e8 name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17917 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17917 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 17921 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@797529c7 name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17922 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17927 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:57008_solr) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6fe865e6 name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17929 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17933 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@265cbc5b name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 17934 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 17934 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>>   [junit4]   2> 17937 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>>   [junit4]   2> 17938 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>>   [junit4]   2> 17946 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>>   [junit4]   2> 17946 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>>   [junit4]   2> 17947 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>>   [junit4]   2> 17948 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>>   [junit4]   2> 17953 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
>>   [junit4]   2> 17960 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
>>   [junit4]   2> 17963 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
>>   [junit4]   2> 17982 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>>   [junit4]   2> 17982 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>>   [junit4]   2> 17983 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>>   [junit4]   2> 17984 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>>   [junit4]   2> 17987 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
>>   [junit4]   2> 17989 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
>>   [junit4]   2> 17991 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
>>   [junit4]   2> 17996 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>>   [junit4]   2> 17997 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>>   [junit4]   2> 17997 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>>   [junit4]   2> 17998 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>>   [junit4]   2> 17998 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>>   [junit4]   2> 17999 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>>   [junit4]   2> 18001 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>>   [junit4]   2> 18001 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>>   [junit4]   2> 18006 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>>   [junit4]   2> 18006 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>>   [junit4]   2> 18007 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>>   [junit4]   2> 18007 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>>   [junit4]   2> 18034 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>>   [junit4]   2> 18034 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>>   [junit4]   2> 18035 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>>   [junit4]   2> 18037 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>>   [junit4]   2> 18038 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
>>   [junit4]   2> 18041 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 18044 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95891707041677321-127.0.0.1:40413_solr-n_0000000000
>>   [junit4]   2> 18043 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 18043 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 18044 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 18045 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95891707041677322-127.0.0.1:63532_solr-n_0000000001
>>   [junit4]   2> 18045 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95891707041677323-127.0.0.1:36992_solr-n_0000000002
>>   [junit4]   2> 18046 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40413_solr
>>   [junit4]   2> 18046 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95891707041677324-127.0.0.1:57008_solr-n_0000000003
>>   [junit4]   2> 18049 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
>>   [junit4]   2> 18050 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95891707041677323-127.0.0.1:36992_solr-n_0000000002 to know if I could be the leader
>>   [junit4]   2> 18050 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95891707041677322-127.0.0.1:63532_solr-n_0000000001 to know if I could be the leader
>>   [junit4]   2> 18050 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57008_solr
>>   [junit4]   2> 18050 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36992_solr
>>   [junit4]   2> 18051 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95891707041677321-127.0.0.1:40413_solr-n_0000000000 to know if I could be the leader
>>   [junit4]   2> 18052 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer Overseer (id=95891707041677321-127.0.0.1:40413_solr-n_0000000000) starting
>>   [junit4]   2> 18053 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63532_solr
>>   [junit4]   2> 18054 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:57008_solr
>>   [junit4]   2> 18054 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:63532_solr
>>   [junit4]   2> 18060 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>>   [junit4]   2> 18061 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:36992_solr
>>   [junit4]   2> 18061 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>>   [junit4]   2> 18063 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:57008_solr) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>>   [junit4]   2> 18063 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>>   [junit4]   2> 18065 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>>   [junit4]   2> 18066 INFO  (zkCallback-24-thread-2-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>>   [junit4]   2> 18069 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
>>   [junit4]   2> 18069 INFO  (zkCallback-24-thread-2-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
>>   [junit4]   2> 18069 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
>>   [junit4]   2> 18070 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:57008_solr) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
>>   [junit4]   2> 18071 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
>>   [junit4]   2> 18073 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
>>   [junit4]   2> 18095 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>>   [junit4]   2> 18096 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>>   [junit4]   2> 18115 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>>   [junit4]   2> 18115 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>>   [junit4]   2> 18144 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
>>   [junit4]   2> 18146 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node1/.
>>   [junit4]   2> 18146 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>>   [junit4]   2> 18148 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1
>>   [junit4]   2> 18148 INFO  (jetty-launcher-6-thread-1) [n:127.0.0.1:57008_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>>   [junit4]   2> 18156 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node3/.
>>   [junit4]   2> 18157 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>>   [junit4]   2> 18158 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1
>>   [junit4]   2> 18158 INFO  (jetty-launcher-6-thread-4) [n:127.0.0.1:36992_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>>   [junit4]   2> 18162 INFO  (OverseerCollectionConfigSetProcessor-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
>>   [junit4]   2> 18164 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:40413_solr as DOWN
>>   [junit4]   2> 18168 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer Starting to work on the main queue
>>   [junit4]   2> 18170 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (3)
>>   [junit4]   2> 18172 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>>   [junit4]   2> 18172 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>>   [junit4]   2> 18173 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/.
>>   [junit4]   2> 18173 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>>   [junit4]   2> 18178 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1
>>   [junit4]   2> 18178 INFO  (jetty-launcher-6-thread-2) [n:127.0.0.1:63532_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>>   [junit4]   2> 18185 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>>   [junit4]   2>   "operation":"downnode",
>>   [junit4]   2>   "node_name":"127.0.0.1:40413_solr"} current state version: 0
>>   [junit4]   2> 18192 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:40413_solr
>>   [junit4]   2> 19171 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40413_solr
>>   [junit4]   2> 19172 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:40413_solr
>>   [junit4]   2> 19184 INFO  (zkCallback-24-thread-2-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>>   [junit4]   2> 19184 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>>   [junit4]   2> 19190 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:57008_solr) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>>   [junit4]   2> 19192 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>>   [junit4]   2> 19192 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>>   [junit4]   2> 19203 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>>   [junit4]   2> 19203 INFO  (zkCallback-24-thread-2-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>>   [junit4]   2> 19203 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:57008_solr) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>>   [junit4]   2> 19209 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>>   [junit4]   2> 19209 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>>   [junit4]   2> 19212 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/.
>>   [junit4]   2> 19213 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>>   [junit4]   2> 19216 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1
>>   [junit4]   2> 19216 INFO  (jetty-launcher-6-thread-3) [n:127.0.0.1:40413_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>>   [junit4]   2> 19217 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 19218 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 19228 INFO  (zkCallback-32-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d1d228d name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 19229 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 19229 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 19237 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:49873/solr
>>   [junit4]   2> 19239 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 19239 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 19259 INFO  (zkCallback-35-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@76a3dba1 name:ZooKeeperConnection Watcher:127.0.0.1:49873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 19259 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 19261 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 19261 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>>   [junit4]   2> 19272 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
>>   [junit4]   2> 19280 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig.xml
>>   [junit4]   2> 19287 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-slave1.xml
>>   [junit4]   2> 19292 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema-replication1.xml
>>   [junit4]   2> 19296 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-managed-schema.xml
>>   [junit4]   2> 19299 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema-sql.xml
>>   [junit4]   2> 19303 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema.xml
>>   [junit4]   2> 19307 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-sql.xml
>>   [junit4]   2> 19345 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWatcherIsRemovedAfterTimeout
>>   [junit4]   2> 19349 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 19350 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 19354 INFO  (zkCallback-36-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c15d6f6 name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 19354 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 19355 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 19374 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [no-such-collection]
>>   [junit4]   2> 19391 INFO  (TEST-TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout-seed#[5F36DD2E780EA866]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWatcherIsRemovedAfterTimeout
>>   [junit4]   2> 19443 INFO  (TEST-TestCollectionStateWatchers.testWaitForStateChecksCurrentState-seed#[5F36DD2E780EA866]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWaitForStateChecksCurrentState
>>   [junit4]   2> 19443 INFO  (TEST-TestCollectionStateWatchers.testWaitForStateChecksCurrentState-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>   [junit4]   2> 19444 INFO  (TEST-TestCollectionStateWatchers.testWaitForStateChecksCurrentState-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>   [junit4]   2> 19448 INFO  (zkCallback-37-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7a20a774 name:ZooKeeperConnection Watcher:127.0.0.1:49873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>   [junit4]   2> 19449 INFO  (TEST-TestCollectionStateWatchers.testWaitForStateChecksCurrentState-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>   [junit4]   2> 19449 INFO  (TEST-TestCollectionStateWatchers.testWaitForStateChecksCurrentState-seed#[5F36DD2E780EA866]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>   [junit4]   2> 19567 INFO  (qtp8715082-95) [n:127.0.0.1:63532_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&name=waitforstate&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
>>   [junit4]   2> 19586 INFO  (OverseerCollectionConfigSetProcessor-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>>   [junit4]   2>   "name":"waitforstate",
>>   [junit4]   2>   "fromApi":"true",
>>   [junit4]   2>   "replicationFactor":"1",
>>   [junit4]   2>   "collection.configName":"config",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "stateFormat":"2",
>>   [junit4]   2>   "operation":"create"}
>>   [junit4]   2> 19588 INFO  (OverseerThreadFactory-36-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
>>   [junit4]   2>   "name":"waitforstate",
>>   [junit4]   2>   "fromApi":"true",
>>   [junit4]   2>   "replicationFactor":"1",
>>   [junit4]   2>   "collection.configName":"config",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "stateFormat":"2",
>>   [junit4]   2>   "operation":"create"}
>>   [junit4]   2> 19590 INFO  (OverseerThreadFactory-36-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerCollectionMessageHandler Create collection waitforstate
>>   [junit4]   2> 19621 INFO  (OverseerThreadFactory-36-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/waitforstate 
>>   [junit4]   2> 19622 INFO  (OverseerThreadFactory-36-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/waitforstate
>>   [junit4]   2> 19631 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:36992_solr) [n:127.0.0.1:36992_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>>   [junit4]   2> 19631 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>>   [junit4]   2> 19631 INFO  (zkCallback-24-thread-2-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>>   [junit4]   2> 19633 INFO  (zkCallback-35-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>>   [junit4]   2> 19633 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:57008_solr) [n:127.0.0.1:57008_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>>   [junit4]   2> 19642 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>>   [junit4]   2>   "name":"waitforstate",
>>   [junit4]   2>   "fromApi":"true",
>>   [junit4]   2>   "replicationFactor":"1",
>>   [junit4]   2>   "collection.configName":"config",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "stateFormat":"2",
>>   [junit4]   2>   "operation":"create"} current state version: 0
>>   [junit4]   2> 19642 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: waitforstate
>>   [junit4]   2> 19669 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/waitforstate/state.json
>>   [junit4]   2> 19745 INFO  (OverseerThreadFactory-36-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating SolrCores for new collection waitforstate, shardNames [shard1] , replicationFactor : 1
>>   [junit4]   2> 19745 INFO  (OverseerThreadFactory-36-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core waitforstate_shard1_replica1 as part of shard shard1 of collection waitforstate on 127.0.0.1:63532_solr
>>   [junit4]   2> 19758 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=config&name=waitforstate_shard1_replica1&action=CREATE&numShards=1&collection=waitforstate&shard=shard1&wt=javabin&version=2
>>   [junit4]   2> 19762 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=waitforstate_shard1_replica1, collection.configName=config, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=waitforstate, numShards=1, dataDir=data/, shard=shard1}
>>   [junit4]   2> 19764 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ZkController publishing state=down
>>   [junit4]   2> 19767 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ZkController Registering watch for collection waitforstate
>>   [junit4]   2> 19768 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.c.ZkStateReader Add data for [waitforstate] ver [0]
>>   [junit4]   2> 19769 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ZkController look for our core node name
>>   [junit4]   2> 19769 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>>   [junit4]   2>   "core":"waitforstate_shard1_replica1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"http://127.0.0.1:63532/solr",
>>   [junit4]   2>   "node_name":"127.0.0.1:63532_solr",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "state":"down",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"waitforstate",
>>   [junit4]   2>   "operation":"state"} current state version: 0
>>   [junit4]   2> 19774 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
>>   [junit4]   2>   "core":"waitforstate_shard1_replica1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"http://127.0.0.1:63532/solr",
>>   [junit4]   2>   "node_name":"127.0.0.1:63532_solr",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "state":"down",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"waitforstate",
>>   [junit4]   2>   "operation":"state"}
>>   [junit4]   2> 19882 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/waitforstate/state.json version: 0
>>   [junit4]   2> 19887 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/waitforstate/state.json] for collection [waitforstate] has occurred - updating... (live nodes size: [4])
>>   [junit4]   2> 19887 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updating data for [waitforstate] from [0] to [1]
>>   [junit4]   2> 20769 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ZkController waiting to find shard id in clusterstate for waitforstate_shard1_replica1
>>   [junit4]   2> 20769 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ZkController Check for collection zkNode:waitforstate
>>   [junit4]   2> 20794 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ZkController Collection zkNode exists
>>   [junit4]   2> 20794 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/waitforstate]
>>   [junit4]   2> 20795 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/waitforstate] [configName]=[config] specified config exists in ZooKeeper
>>   [junit4]   2> 20796 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/waitforstate_shard1_replica1'
>>   [junit4]   2> 20796 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>   [junit4]   2> 20796 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>>   [junit4]   2> 20799 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
>>   [junit4]   2> 20832 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.SolrConfig current version of requestparams : -1
>>   [junit4]   2> 20840 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
>>   [junit4]   2> 20868 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
>>   [junit4]   2> 20886 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.s.IndexSchema [waitforstate_shard1_replica1] Schema name=test
>>   [junit4]   2> 21108 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.s.IndexSchema default search field in schema is text
>>   [junit4]   2> 21109 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.s.IndexSchema unique key field: id
>>   [junit4]   2> 21109 WARN  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
>>   [junit4]   2> 21140 WARN  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
>>   [junit4]   2> 21199 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/config', cwd=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1
>>   [junit4]   2> 21200 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1  x:waitforstate_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'waitforstate_shard1_replica1' using configuration from collection waitforstate
>>   [junit4]   2> 21200 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
>>   [junit4]   2> 21201 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore [[waitforstate_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/waitforstate_shard1_replica1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data/]
>>   [junit4]   2> 21202 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore JMX monitoring not detected for core: waitforstate_shard1_replica1
>>   [junit4]   2> 21205 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data
>>   [junit4]   2> 21205 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data/index/
>>   [junit4]   2> 21205 WARN  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore [waitforstate_shard1_replica1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data/index' doesn't exist. Creating new index...
>>   [junit4]   2> 21205 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data/index
>>   [junit4]   2> 21206 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
>>   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3735e084 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@69bd3a6e),segFN=segments_1,generation=1}
>>   [junit4]   2> 21206 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>>   [junit4]   2> 21211 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
>>   [junit4]   2> 21214 INFO  (OldIndexDirectoryCleanupThreadForCore-waitforstate_shard1_replica1) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core waitforstate_shard1_replica1 in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data/
>>   [junit4]   2> 21215 WARN  (OldIndexDirectoryCleanupThreadForCore-waitforstate_shard1_replica1) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.DirectoryFactory /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node2/./waitforstate_shard1_replica1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
>>   [junit4]   2> 21218 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 21220 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 21254 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 21255 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 21268 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file
>>   [junit4]   2> 21268 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
>>   [junit4]   2> 21269 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
>>   [junit4]   2> 21269 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>>   [junit4]   2> 21295 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
>>   [junit4]   2> 21295 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
>>   [junit4]   2> 21296 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
>>   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3735e084 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@69bd3a6e),segFN=segments_1,generation=1}
>>   [junit4]   2> 21296 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>>   [junit4]   2> 21296 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3137bdbb[waitforstate_shard1_replica1] main]
>>   [junit4]   2> 21296 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/waitforstate]
>>   [junit4]   2> 21298 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/waitforstate] [configName]=[config] specified config exists in ZooKeeper
>>   [junit4]   2> 21298 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/config
>>   [junit4]   2> 21312 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
>>   [junit4]   2> 21312 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
>>   [junit4]   2> 21312 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
>>   [junit4]   2> 21313 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/config/_rest_managed.json
>>   [junit4]   2> 21315 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
>>   [junit4]   2> 21315 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
>>   [junit4]   2> 21315 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
>>   [junit4]   2> 21337 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
>>   [junit4]   2> 21337 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
>>   [junit4]   2> 21337 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
>>   [junit4]   2> 21337 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1534267318663643136
>>   [junit4]   2> 21340 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1534267318663643136
>>   [junit4]   2> 21342 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ZkController watch zkdir /configs/config
>>   [junit4]   2> 21344 INFO  (searcherExecutor-39-thread-1-processing-n:127.0.0.1:63532_solr x:waitforstate_shard1_replica1 s:shard1 c:waitforstate r:core_node1) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SolrCore [waitforstate_shard1_replica1] Registered new searcher Searcher@3137bdbb[waitforstate_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>   [junit4]   2> 21345 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.CoreContainer registering core: waitforstate_shard1_replica1
>>   [junit4]   2> 21345 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ZkController Register replica - core:waitforstate_shard1_replica1 address:http://127.0.0.1:63532/solr collection:waitforstate shard:shard1
>>   [junit4]   2> 21364 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.c.SolrZkClient makePath: /collections/waitforstate/leader_elect/shard1/election
>>   [junit4]   2> 21408 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/waitforstate/leader_elect/shard1/election/95891707041677322-core_node1-n_0000000000
>>   [junit4]   2> 21409 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
>>   [junit4]   2> 21414 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>   [junit4]   2> 21414 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
>>   [junit4]   2> 21416 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:63532/solr/waitforstate_shard1_replica1/
>>   [junit4]   2> 21416 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>   [junit4]   2> 21416 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.SyncStrategy http://127.0.0.1:63532/solr/waitforstate_shard1_replica1/ has no replicas
>>   [junit4]   2> 21418 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>>   [junit4]   2>   "operation":"leader",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"waitforstate"} current state version: 0
>>   [junit4]   2> 21426 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/waitforstate/state.json version: 1
>>   [junit4]   2> 21429 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/waitforstate/state.json] for collection [waitforstate] has occurred - updating... (live nodes size: [4])
>>   [junit4]   2> 21431 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.c.SolrZkClient makePath: /collections/waitforstate/leaders/shard1
>>   [junit4]   2> 21432 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updating data for [waitforstate] from [1] to [2]
>>   [junit4]   2> 21460 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/waitforstate/leaders/shard1/leader after winning as /collections/waitforstate/leader_elect/shard1/election/95891707041677322-core_node1-n_0000000000
>>   [junit4]   2> 21530 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:63532/solr/waitforstate_shard1_replica1/ shard1
>>   [junit4]   2> 21532 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>>   [junit4]   2>   "operation":"leader",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"waitforstate",
>>   [junit4]   2>   "base_url":"http://127.0.0.1:63532/solr",
>>   [junit4]   2>   "core":"waitforstate_shard1_replica1",
>>   [junit4]   2>   "state":"active"} current state version: 0
>>   [junit4]   2> 21639 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/waitforstate/state.json version: 2
>>   [junit4]   2> 21640 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/waitforstate/state.json] for collection [waitforstate] has occurred - updating... (live nodes size: [4])
>>   [junit4]   2> 21642 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:63532_solr) [n:127.0.0.1:63532_solr    ] o.a.s.c.c.ZkStateReader Updating data for [waitforstate] from [2] to [3]
>>   [junit4]   2> 21683 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ZkController We are http://127.0.0.1:63532/solr/waitforstate_shard1_replica1/ and leader is http://127.0.0.1:63532/solr/waitforstate_shard1_replica1/
>>   [junit4]   2> 21684 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ZkController No LogReplay needed for core=waitforstate_shard1_replica1 baseURL=http://127.0.0.1:63532/solr
>>   [junit4]   2> 21684 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
>>   [junit4]   2> 21684 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.c.ZkController publishing state=active
>>   [junit4]   2> 21689 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>>   [junit4]   2>   "core":"waitforstate_shard1_replica1",
>>   [junit4]   2>   "core_node_name":"core_node1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"http://127.0.0.1:63532/solr",
>>   [junit4]   2>   "node_name":"127.0.0.1:63532_solr",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "state":"active",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"waitforstate",
>>   [junit4]   2>   "operation":"state"} current state version: 0
>>   [junit4]   2> 21691 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
>>   [junit4]   2>   "core":"waitforstate_shard1_replica1",
>>   [junit4]   2>   "core_node_name":"core_node1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"http://127.0.0.1:63532/solr",
>>   [junit4]   2>   "node_name":"127.0.0.1:63532_solr",
>>   [junit4]   2>   "numShards":"1",
>>   [junit4]   2>   "state":"active",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"waitforstate",
>>   [junit4]   2>   "operation":"state"}
>>   [junit4]   2> 21692 INFO  (qtp8715082-94) [n:127.0.0.1:63532_solr c:waitforstate s:shard1 r:core_node1 x:waitforstate_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=config&name=waitforstate_shard
>> 
>> [...truncated too long message...]
>> 
>> llection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.SolrCore [testcollection_shard2_replica1] Closing main searcher on request.
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./testcollection_shard2_replica1/data/index [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./testcollection_shard2_replica1/data/index;done=false>>]
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./testcollection_shard2_replica1/data/index
>>   [junit4]   2> 47048 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testcollection/state.json version: 5
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-1) [n:127.0.0.1:40413_solr c:falsepredicate s:shard1 r:core_node4 x:falsepredicate_shard1_replica1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./falsepredicate_shard1_replica1/data/index
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./testcollection_shard2_replica1/data [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./testcollection_shard2_replica1/data;done=false>>]
>>   [junit4]   2> 47048 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
>>   [junit4]   2> 47049 INFO  (coreCloseExecutor-143-thread-2) [n:127.0.0.1:40413_solr c:testcollection s:shard2 r:core_node4 x:testcollection_shard2_replica1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001/tempDir-001/node4/./testcollection_shard2_replica1/data
>>   [junit4]   2> 47050 INFO  (zkCallback-24-thread-3-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [1])
>>   [junit4]   2> 47053 INFO  (jetty-launcher-6-thread-2) [    ] o.a.s.c.Overseer Overseer (id=95891707041677321-127.0.0.1:40413_solr-n_0000000000) closing
>>   [junit4]   2> 47054 INFO  (OverseerStateUpdate-95891707041677321-127.0.0.1:40413_solr-n_0000000000) [n:127.0.0.1:40413_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40413_solr
>>   [junit4]   2> 47055 INFO  (zkCallback-24-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
>>   [junit4]   2> 47056 INFO  (zkCallback-24-thread-3-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader Updating data for [testcollection] from [5] to [6]
>>   [junit4]   2> 48565 WARN  (zkCallback-24-thread-1-processing-n:127.0.0.1:40413_solr) [n:127.0.0.1:40413_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
>>   [junit4]   2> 48566 INFO  (jetty-launcher-6-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@34f50864{/solr,null,UNAVAILABLE}
>>   [junit4]   2> 48571 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49873 49873
>>   [junit4]   2> 48587 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
>>   [junit4]   2> java.nio.channels.CancelledKeyException
>>   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>>   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
>>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
>>   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
>>   [junit4]   2> 48588 INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49873 49873
>>   [junit4]   2> 48649 WARN  (Thread-14) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
>>   [junit4]   2> Maximum concurrent create/delete watches above limit:
>>   [junit4]   2> 
>>   [junit4]   2> 	8	/solr/aliases.json
>>   [junit4]   2> 	7	/solr/security.json
>>   [junit4]   2> 	7	/solr/configs/config
>>   [junit4]   2> 	3	/solr/collections/stateformat1/state.json
>>   [junit4]   2> 
>>   [junit4]   2> Maximum concurrent data watches above limit:
>>   [junit4]   2> 
>>   [junit4]   2> 	12	/solr/collections/waitforstate/state.json
>>   [junit4]   2> 	8	/solr/clusterstate.json
>>   [junit4]   2> 	8	/solr/collections/testcollection/state.json
>>   [junit4]   2> 	7	/solr/collections/falsepredicate/state.json
>>   [junit4]   2> 	3	/solr/overseer_elect/election/95891707041677323-127.0.0.1:36992_solr-n_0000000002
>>   [junit4]   2> 	3	/solr/collections/stateformat1/state.json
>>   [junit4]   2> 	2	/solr/overseer_elect/election/95891707041677321-127.0.0.1:40413_solr-n_0000000000
>>   [junit4]   2> 	2	/solr/collections/delayed/state.json
>>   [junit4]   2> 	2	/solr/collections/tobedeleted/state.json
>>   [junit4]   2> 
>>   [junit4]   2> Maximum concurrent children watches above limit:
>>   [junit4]   2> 
>>   [junit4]   2> 	8	/solr/collections
>>   [junit4]   2> 	5	/solr/live_nodes
>>   [junit4]   2> 
>>   [junit4]   2> 48650 INFO  (SUITE-TestCollectionStateWatchers-seed#[5F36DD2E780EA866]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
>>   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.common.cloud.TestCollectionStateWatchers_5F36DD2E780EA866-001
>>   [junit4]   2> May 14, 2016 2:02:43 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>>   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
>>   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=340, maxMBSortInHeap=5.006846218720272, sim=ClassicSimilarity, locale=en-PH, timezone=America/Anchorage
>>   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=1,free=137478512,total=279969792
>>   [junit4]   2> NOTE: All tests run in this JVM: [JdbcDriverTest, HttpClientUtilTest, LBHttpSolrClientBuilderTest, TestUpdateRequestCodec, BasicHttpSolrClientTest, SolrQueryTest, ShardParamsTest, ConcatOperationTest, TestToleratedUpdateError, QueryResponseTest, CloudSolrClientMultiConstructorTest, TestCollectionStateWatchers]
>>   [junit4] Completed [22/85 (1!)] on J1 in 33.75s, 8 tests, 1 failure <<< FAILURES!
>> 
>> [...truncated 199 lines...]
>> BUILD FAILED
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:740: The following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:684: The following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:59: The following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build.xml:246: The following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/common-build.xml:529: The following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:1457: The following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:1014: There were test failures: 85 suites, 646 tests, 1 failure, 2 ignored [seed: 5F36DD2E780EA866]
>> 
>> Total time: 84 minutes 45 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> [WARNINGS] Skipping publisher since build result is FAILURE
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
> 


Mime
View raw message