lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-master - Build # 967 - Still Failing
Date Mon, 21 Mar 2016 15:44:16 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/967/

2 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=23723, name=testExecutor-3881-thread-6, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=23723, name=testExecutor-3881-thread-6, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60272/nx/gd
	at __randomizedtesting.SeedInfo.seed([7BC66ACCDDF54135]:0)
	at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:583)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60272/nx/gd
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:581)
	... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
	... 8 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 3 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 3 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([7BC66ACCDDF54135]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:238)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11738 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/init-core-data-001
   [junit4]   2> 1543496 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /nx/gd
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1543554 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1543563 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1543574 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1543588 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_42076_hdfs____.b7rypx/webapp
   [junit4]   2> 1543721 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1543970 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42076
   [junit4]   2> 1544056 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1544057 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1544073 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40884_datanode____.frg7a9/webapp
   [junit4]   2> 1544193 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1544475 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40884
   [junit4]   2> 1544538 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1544539 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1544559 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45109_datanode____n7qog0/webapp
   [junit4]   2> 1544631 INFO  (IPC Server handler 6 on 34977) [    ] BlockStateChange BLOCK* processReport: from storage DS-15185ce1-b55f-4f13-9e89-c0d7ce393bf5 node DatanodeRegistration(127.0.0.1, datanodeUuid=2b747ff9-f796-40d6-bfba-9d7e105227cc, infoPort=40884, ipcPort=52730, storageInfo=lv=-56;cid=testClusterID;nsid=818913130;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1544631 INFO  (IPC Server handler 6 on 34977) [    ] BlockStateChange BLOCK* processReport: from storage DS-0e7da3d3-fd3f-45ef-a353-ecd099475f50 node DatanodeRegistration(127.0.0.1, datanodeUuid=2b747ff9-f796-40d6-bfba-9d7e105227cc, infoPort=40884, ipcPort=52730, storageInfo=lv=-56;cid=testClusterID;nsid=818913130;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1544682 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1544925 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[7BC66ACCDDF54135]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45109
   [junit4]   2> 1545119 INFO  (IPC Server handler 6 on 34977) [    ] BlockStateChange BLOCK* processReport: from storage DS-763eef36-3ab3-4cb2-a0f4-3d08174f5890 node DatanodeRegistration(127.0.0.1, datanodeUuid=3bfc074f-f6be-43ab-8f5a-6b3923c0f14a, infoPort=45109, ipcPort=55416, storageInfo=lv=-56;cid=testClusterID;nsid=818913130;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 1545119 INFO  (IPC Server handler 6 on 34977) [    ] BlockStateChange BLOCK* processReport: from storage DS-1e545f7d-071b-44da-851d-59bbe9333c3c node DatanodeRegistration(127.0.0.1, datanodeUuid=3bfc074f-f6be-43ab-8f5a-6b3923c0f14a, infoPort=45109, ipcPort=55416, storageInfo=lv=-56;cid=testClusterID;nsid=818913130;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1545240 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1545240 INFO  (Thread-14273) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1545240 INFO  (Thread-14273) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1545340 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ZkTestServer start zk server on port:60880
   [junit4]   2> 1545340 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1545345 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1545351 INFO  (zkCallback-1350-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@64ceefcd name:ZooKeeperConnection Watcher:127.0.0.1:60880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1545351 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1545351 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1545351 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1545354 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1545355 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1545356 INFO  (zkCallback-1351-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6e10bbaf name:ZooKeeperConnection Watcher:127.0.0.1:60880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1545356 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1545356 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1545356 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1545358 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1545359 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1545360 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1545361 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1545361 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1545363 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1545363 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1545365 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1545365 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1545366 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1545366 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1545367 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1545367 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1545368 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1545369 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1545370 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1545370 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1545371 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1545371 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1545372 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1545372 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1545374 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1545374 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1545375 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1545375 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1545497 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores/collection1
   [junit4]   2> 1545498 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1545500 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32327968{/nx/gd,null,AVAILABLE}
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.ServerConnector Started ServerConnector@23404bf2{HTTP/1.1,[http/1.1]}{127.0.0.1:49607}
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.Server Started @1548854ms
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:34977/hdfs__localhost_34977__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001_tempDir-002_control_data, hostContext=/nx/gd, hostPort=49607, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores}
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001'
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1545501 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1545502 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1545503 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1545503 INFO  (zkCallback-1352-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5bf90a03 name:ZooKeeperConnection Watcher:127.0.0.1:60880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1545503 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1545504 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1545506 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1545506 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/solr.xml
   [junit4]   2> 1545512 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores
   [junit4]   2> 1545512 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CoreContainer New CoreContainer 1737857786
   [junit4]   2> 1545512 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001]
   [junit4]   2> 1545512 WARN  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/lib
   [junit4]   2> 1545513 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1545514 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1545515 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1545515 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1545515 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60880/solr
   [junit4]   2> 1545515 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1545515 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1545516 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1545517 INFO  (zkCallback-1355-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@29e25574 name:ZooKeeperConnection Watcher:127.0.0.1:60880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1545517 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1545518 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1545519 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1545520 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46d5e01 name:ZooKeeperConnection Watcher:127.0.0.1:60880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1545520 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1545521 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1545523 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1545528 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1545531 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1545536 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1545538 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1545539 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1545541 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1545542 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1545543 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1545545 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1545548 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1545549 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1545551 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1545552 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000
   [junit4]   2> 1545553 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49607_nx%2Fgd
   [junit4]   2> 1545553 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1545555 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer Overseer (id=95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) starting
   [junit4]   2> 1545557 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1545568 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1545568 INFO  (OverseerCollectionConfigSetProcessor-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1545569 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1545570 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49607_nx%2Fgd
   [junit4]   2> 1545572 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:49607_nx%2Fgd
   [junit4]   2> 1545574 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1545576 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1545576 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 1545589 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1545589 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1545590 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores
   [junit4]   2> 1545591 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1545591 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores/collection1
   [junit4]   2> 1545591 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1545593 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1545593 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0
   [junit4]   2> 1545593 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1545593 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1545595 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49607/nx/gd",
   [junit4]   2>   "node_name":"127.0.0.1:49607_nx%2Fgd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1545595 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1545595 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49607/nx/gd",
   [junit4]   2>   "node_name":"127.0.0.1:49607_nx%2Fgd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1545595 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1545596 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1545597 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1546595 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1546595 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1546596 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1546596 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1546596 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1546596 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores/collection1'
   [junit4]   2> 1546596 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1546597 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1546598 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1546603 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1546609 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1546620 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1546625 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1546752 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1546758 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1546760 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1546768 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1546771 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1546781 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1546785 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1546789 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0
   [junit4]   2> 1546789 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1546789 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 1546790 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:34977/solr_hdfs_home
   [junit4]   2> 1546790 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1546791 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/control-001/cores/collection1], dataDir=[hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1546791 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3269e027
   [junit4]   2> 1546792 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1546814 WARN  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 1551815 WARN  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 1556820 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1556822 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 1556831 WARN  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:34977/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1556831 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1556838 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1556838 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1556838 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1556848 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1556848 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1556849 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6454272233011404]
   [junit4]   2> 1556865 INFO  (IPC Server handler 4 on 34977) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51544 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-1e545f7d-071b-44da-851d-59bbe9333c3c:NORMAL:127.0.0.1:57258|RBW], ReplicaUnderConstruction[[DISK]DS-0e7da3d3-fd3f-45ef-a353-ecd099475f50:NORMAL:127.0.0.1:51544|RBW]]} size 0
   [junit4]   2> 1556867 INFO  (IPC Server handler 8 on 34977) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57258 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e7da3d3-fd3f-45ef-a353-ecd099475f50:NORMAL:127.0.0.1:51544|RBW], ReplicaUnderConstruction[[DISK]DS-763eef36-3ab3-4cb2-a0f4-3d08174f5890:NORMAL:127.0.0.1:57258|FINALIZED]]} size 0
   [junit4]   2> 1556869 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@77f3c510); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1556869 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1556870 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 1556873 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1556874 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1556874 WARN  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1556876 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1556877 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1556878 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1556879 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1556882 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1556882 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1556883 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1556883 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 1556893 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1556893 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1556894 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1556898 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:34977/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@77f3c510); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1556898 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1556898 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@53ae2123[collection1] main]
   [junit4]   2> 1556898 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1556899 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1556899 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1556899 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1556899 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1556899 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1556900 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1556900 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1556900 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1556900 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1556901 INFO  (searcherExecutor-3803-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@53ae2123[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1556908 WARN  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Could not retrieve locality information for hdfs://localhost:54637/solr3 due to exception: java.net.ConnectException: Call From lucene1-us-west/10.41.0.5 to localhost:54637 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
   [junit4]   2> 1556908 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1556908 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 1556908 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1556909 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1529423529379889152
   [junit4]   2> 1556909 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with highest version 1529423529379889152
   [junit4]   2> 1556909 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1556909 INFO  (coreLoadExecutor-3802-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1556910 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:49607/nx/gd collection:control_collection shard:shard1
   [junit4]   2> 1556910 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1556913 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95588969821569028-core_node1-n_0000000000
   [junit4]   2> 1556913 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1556915 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1556915 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1556915 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:49607/nx/gd/collection1/
   [junit4]   2> 1556915 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1556915 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:49607/nx/gd/collection1/ has no replicas
   [junit4]   2> 1556915 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1556916 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1556916 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1556919 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95588969821569028-core_node1-n_0000000000
   [junit4]   2> 1556920 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:49607/nx/gd/collection1/ shard1
   [junit4]   2> 1556921 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:49607/nx/gd",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1557023 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1557072 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:49607/nx/gd/collection1/ and leader is http://127.0.0.1:49607/nx/gd/collection1/
   [junit4]   2> 1557072 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49607/nx/gd
   [junit4]   2> 1557072 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1557072 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1557072 INFO  (coreZkRegister-3796-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49607_nx%2Fgd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1557073 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49607/nx/gd",
   [junit4]   2>   "node_name":"127.0.0.1:49607_nx%2Fgd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1557074 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49607/nx/gd",
   [junit4]   2>   "node_name":"127.0.0.1:49607_nx%2Fgd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1557097 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1557098 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557099 INFO  (zkCallback-1358-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@174e6e4f name:ZooKeeperConnection Watcher:127.0.0.1:60880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557099 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557099 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1557099 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1557100 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1557101 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1557176 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1557176 INFO  (zkCallback-1358-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1557206 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001/cores/collection1
   [junit4]   2> 1557207 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001
   [junit4]   2> 1557208 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1557211 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@57b6b225{/nx/gd,null,AVAILABLE}
   [junit4]   2> 1557211 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6dc0153b{HTTP/1.1,[http/1.1]}{127.0.0.1:47590}
   [junit4]   2> 1557211 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.Server Started @1560565ms
   [junit4]   2> 1557211 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:34977/hdfs__localhost_34977__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/nx/gd, hostPort=47590, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001/cores}
   [junit4]   2> 1557211 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1557212 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001'
   [junit4]   2> 1557212 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1557212 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1557212 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1557213 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557214 INFO  (zkCallback-1359-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ed235cd name:ZooKeeperConnection Watcher:127.0.0.1:60880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557214 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557214 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1557215 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1557215 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001/solr.xml
   [junit4]   2> 1557221 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001/cores
   [junit4]   2> 1557221 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CoreContainer New CoreContainer 1712575957
   [junit4]   2> 1557221 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001]
   [junit4]   2> 1557222 WARN  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_7BC66ACCDDF54135-001/shard-1-001/lib
   [junit4]   2> 1557222 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1557223 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1557223 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1557224 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1557224 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60880/solr
   [junit4]   2> 1557224 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1557224 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1557224 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557225 INFO  (zkCallback-1362-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2f8f33d4 name:ZooKeeperConnection Watcher:127.0.0.1:60880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557225 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557226 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1557231 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557231 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:47590_nx%2Fgd) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@209558a1 name:ZooKeeperConnection Watcher:127.0.0.1:60880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557231 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557239 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1557241 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1557243 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1557244 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95588969821569032-127.0.0.1:47590_nx%2Fgd-n_0000000001
   [junit4]   2> 1557244 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000 to know if I could be the leader
   [junit4]   2> 1557245 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.ZkController Publish node=127.0.0.1:47590_nx%2Fgd as DOWN
   [junit4]   2> 1557246 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:47590_nx%2Fgd"} current state version: 4
   [junit4]   2> 1557246 INFO  (OverseerStateUpdate-95588969821569028-127.0.0.1:49607_nx%2Fgd-n_0000000000) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:47590_nx%2Fgd
   [junit4]   2> 1557349 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1557349 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:47590_nx%2Fgd) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1557349 INFO  (zkCallback-1358-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1558246 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47590_nx%2Fgd
   [junit4]   2> 1558247 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[7BC66ACCDDF54135]) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:47590_nx%2Fgd
   [junit4]   2> 1558248 INFO  (zkCallback-1356-thread-1-processing-n:127.0.0.1:49607_nx%2Fgd) [n:127.0.0.1:49607_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1558248 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:47590_nx%2Fgd) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1558248 INFO  (zkCallback-1358-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1558249 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:47590_nx%2Fgd) [n:127.0.0.1:47590_nx%2Fgd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1558249 INFO  (zkCallback-1358-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)

[...truncated too long message...]

Executor-4871-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2413200 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2413200 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 2413200 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2413200 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7BC66ACCDDF54135-001/solr-instance-027/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7BC66ACCDDF54135-001/solr-instance-027/./collection1/data/index;done=false>>]
   [junit4]   2> 2413200 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7BC66ACCDDF54135-001/solr-instance-027/./collection1/data/index
   [junit4]   2> 2413201 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7BC66ACCDDF54135-001/solr-instance-027/./collection1/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7BC66ACCDDF54135-001/solr-instance-027/./collection1/data;done=false>>]
   [junit4]   2> 2413201 INFO  (coreCloseExecutor-4871-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7BC66ACCDDF54135-001/solr-instance-027/./collection1/data
   [junit4]   2> 2413201 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[7BC66ACCDDF54135]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6fc2ede2{/solr,null,UNAVAILABLE}
   [junit4]   2> 2413203 INFO  (SUITE-TestReplicationHandler-seed#[7BC66ACCDDF54135]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> ObjectTracker found 3 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {name=PostingsFormat(name=Direct), id=PostingsFormat(name=Direct), newname=PostingsFormat(name=Direct)}, docValues:{}, maxPointsInLeafNode=1146, maxMBSortInHeap=5.5987099031934005, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ar-AE, timezone=America/Santa_Isabel
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=224232312,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestPseudoReturnFields, SystemInfoHandlerTest, TestFaceting, JSONWriterTest, TestSolrConfigHandlerConcurrent, TestBadConfig, PingRequestHandlerTest, TestMergePolicyConfig, DistribCursorPagingTest, CollectionTooManyReplicasTest, TestJsonFacets, SOLR749Test, DisMaxRequestHandlerTest, DistributedQueryComponentOptimizationTest, SaslZkACLProviderTest, BadCopyFieldTest, XsltUpdateRequestHandlerTest, SparseHLLTest, ActionThrottleTest, TestAnalyzedSuggestions, TestCloudSchemaless, ReturnFieldsTest, TestHdfsUpdateLog, SynonymTokenizerTest, HdfsSyncSliceTest, TestSchemaManager, TestHighFrequencyDictionaryFactory, TestSolr4Spatial, DocExpirationUpdateProcessorFactoryTest, TestSolrXml, FileBasedSpellCheckerTest, StandardRequestHandlerTest, SharedFSAutoReplicaFailoverUtilsTest, CoreMergeIndexesAdminHandlerTest, MergeStrategyTest, SpatialHeatmapFacetsTest, TestSerializedLuceneMatchVersion, TestRandomFaceting, TestJoin, AtomicUpdatesTest, DataDrivenBlockJoinTest, SyncSliceTest, AlternateDirectoryTest, TestCSVLoader, TestFilteredDocIdSet, SolrXmlInZkTest, TestAddFieldRealTimeGet, TestScoreJoinQPNoScore, HttpPartitionTest, SignatureUpdateProcessorFactoryTest, TestFastOutputStream, TestDFISimilarityFactory, RequiredFieldsTest, TestMiniSolrCloudCluster, OutputWriterTest, PathHierarchyTokenizerFactoryTest, TestSchemaResource, TestFoldingMultitermQuery, AnalyticsMergeStrategyTest, TestLuceneMatchVersion, TestTrackingShardHandlerFactory, ResponseLogComponentTest, CursorPagingTest, DocumentAnalysisRequestHandlerTest, TestReloadDeadlock, PrimUtilsTest, BasicZkTest, TestIBSimilarityFactory, DeleteReplicaTest, TestInitParams, AddBlockUpdateTest, ConcurrentDeleteAndCreateCollectionTest, DocValuesMissingTest, IndexSchemaRuntimeFieldTest, HdfsBasicDistributedZk2Test, HdfsDirectoryFactoryTest, TestCryptoKeys, TestRandomDVFaceting, HdfsUnloadDistributedZkTest, PolyFieldTest, TestOmitPositions, TestDocSet, TestReplicationHandler]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.seed=7BC66ACCDDF54135 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar-AE -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestReplicationHandler (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 3 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7BC66ACCDDF54135]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:238)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [283/580 (2!)] on J0 in 699.85s, 13 tests, 1 failure <<< FAILURES!

[...truncated 940 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1014: There were test failures: 580 suites (10 ignored), 2323 tests, 1 suite-level error, 1 error, 32 ignored (16 assumptions) [seed: 7BC66ACCDDF54135]

Total time: 205 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message