lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b15) - Build # 10279 - Still Failing!
Date Sun, 11 May 2014 12:20:36 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10279/
Java: 32bit/jdk1.7.0_60-ea-b15 -client -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
createcollection the collection error [Watcher fired on path: null state: SyncConnected type None]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: createcollection the collection error [Watcher fired on path: null state: SyncConnected type None]
	at __randomizedtesting.SeedInfo.seed([C611100F7AC1DEA3:47F79E170D9EBE9F]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.CollectionAdminRequest.process(CollectionAdminRequest.java:300)
	at org.apache.solr.client.solrj.request.CollectionAdminRequest.createCollection(CollectionAdminRequest.java:360)
	at org.apache.solr.client.solrj.request.CollectionAdminRequest.createCollection(CollectionAdminRequest.java:346)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSolrJAPICalls(CollectionsAPIDistributedZkTest.java:298)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:201)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11138 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/init-core-data-001
   [junit4]   2> 747292 T3280 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 747293 T3280 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /zjh/
   [junit4]   2> 747298 T3280 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 747298 T3280 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 747299 T3281 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 747399 T3280 oasc.ZkTestServer.run start zk server on port:37831
   [junit4]   2> 747400 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753549 T3284 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 6147ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 753551 T3287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fd7cb1 name:ZooKeeperConnection Watcher:127.0.0.1:37831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753551 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 753551 T3280 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 753554 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753555 T3289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bc9ab4 name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753556 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 753556 T3280 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 753567 T3280 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 753569 T3280 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 753570 T3280 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 753572 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 753572 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 753580 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 753581 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 753585 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 753585 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 753589 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 753591 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 753594 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 753594 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 753598 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 753599 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 753604 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 753604 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 753606 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 753607 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 753609 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 753609 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 753611 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 753612 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 753614 T3280 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 753615 T3280 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 753617 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753618 T3291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@126758f name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753618 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 753923 T3280 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 753925 T3280 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 753926 T3280 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40582
   [junit4]   2> 753928 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 753928 T3280 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 753929 T3280 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002
   [junit4]   2> 753929 T3280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/'
   [junit4]   2> 753942 T3280 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/solr.xml
   [junit4]   2> 753972 T3280 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002
   [junit4]   2> 753972 T3280 oasc.CoreContainer.<init> New CoreContainer 1870730
   [junit4]   2> 753972 T3280 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/]
   [junit4]   2> 753973 T3280 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 753974 T3280 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 753974 T3280 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 753974 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 753975 T3280 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 753975 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 753975 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 753975 T3280 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 753976 T3280 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 753978 T3280 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 753978 T3280 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 753978 T3280 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 753979 T3280 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37831/solr
   [junit4]   2> 753979 T3280 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 753980 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753982 T3302 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17042d6 name:ZooKeeperConnection Watcher:127.0.0.1:37831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753982 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 753984 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753985 T3304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19878b9 name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753985 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 753987 T3280 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 753989 T3280 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 753991 T3280 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 753992 T3280 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 753994 T3280 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 753996 T3280 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 753997 T3280 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40582_zjh
   [junit4]   2> 753998 T3280 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40582_zjh
   [junit4]   2> 754002 T3280 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 754003 T3280 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 754005 T3280 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40582_zjh
   [junit4]   2> 754006 T3280 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 754007 T3280 oasc.Overseer.start Overseer (id=91737959269072900-127.0.0.1:40582_zjh-n_0000000000) starting
   [junit4]   2> 754011 T3280 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 754018 T3306 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 754019 T3280 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 754019 T3306 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:40582_zjh
   [junit4]   2> 754020 T3280 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 754021 T3280 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 754023 T3305 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 754025 T3280 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002
   [junit4]   2> 754030 T3280 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/conf/
   [junit4]   2> 754130 T3280 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 754132 T3280 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 754132 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 754134 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 754135 T3309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@146803f name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 754135 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 754137 T3280 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 754138 T3280 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 754381 T3280 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 754382 T3280 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 754384 T3280 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 754385 T3280 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43091
   [junit4]   2> 754387 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 754387 T3280 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 754387 T3280 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003
   [junit4]   2> 754388 T3280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/'
   [junit4]   2> 754401 T3280 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solr.xml
   [junit4]   2> 754430 T3280 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003
   [junit4]   2> 754430 T3280 oasc.CoreContainer.<init> New CoreContainer 15810289
   [junit4]   2> 754431 T3280 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/]
   [junit4]   2> 754432 T3280 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 754432 T3280 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 754432 T3280 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 754432 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 754433 T3280 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 754433 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 754433 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 754434 T3280 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 754434 T3280 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 754436 T3280 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 754436 T3280 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 754437 T3280 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 754437 T3280 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37831/solr
   [junit4]   2> 754437 T3280 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 754438 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 754439 T3320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121c485 name:ZooKeeperConnection Watcher:127.0.0.1:37831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 754440 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 754442 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 754442 T3322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1427c3e name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 754443 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 754447 T3280 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 755449 T3280 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43091_zjh
   [junit4]   2> 755451 T3280 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43091_zjh
   [junit4]   2> 755457 T3280 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003
   [junit4]   2> 755463 T3280 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/conf/
   [junit4]   2> 755609 T3280 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 755610 T3280 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 755611 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 755852 T3280 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 755852 T3280 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 755855 T3280 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 755856 T3280 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53652
   [junit4]   2> 755858 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 755858 T3280 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 755858 T3280 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004
   [junit4]   2> 755859 T3280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/'
   [junit4]   2> 755874 T3280 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solr.xml
   [junit4]   2> 755904 T3280 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004
   [junit4]   2> 755904 T3280 oasc.CoreContainer.<init> New CoreContainer 14542218
   [junit4]   2> 755904 T3280 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/]
   [junit4]   2> 755905 T3280 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 755906 T3280 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 755906 T3280 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 755907 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 755907 T3280 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 755908 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 755908 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 755909 T3280 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 755909 T3280 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 755911 T3280 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 755911 T3280 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 755912 T3280 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 755912 T3280 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37831/solr
   [junit4]   2> 755912 T3280 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 755913 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 755915 T3334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b0dc15 name:ZooKeeperConnection Watcher:127.0.0.1:37831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 755915 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 755917 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 755920 T3336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e59995 name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 755920 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 755926 T3280 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 756928 T3280 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53652_zjh
   [junit4]   2> 756934 T3280 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53652_zjh
   [junit4]   2> 756942 T3280 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004
   [junit4]   2> 756953 T3280 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/conf/
   [junit4]   2> 757180 T3280 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 757182 T3280 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 757182 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 757452 T3280 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 757452 T3280 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 757455 T3280 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 757457 T3280 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:54306
   [junit4]   2> 757459 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 757459 T3280 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 757460 T3280 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005
   [junit4]   2> 757460 T3280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005/'
   [junit4]   2> 757477 T3280 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005/solr.xml
   [junit4]   2> 757508 T3280 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005
   [junit4]   2> 757509 T3280 oasc.CoreContainer.<init> New CoreContainer 29396227
   [junit4]   2> 757509 T3280 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005/]
   [junit4]   2> 757510 T3280 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 757510 T3280 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 757511 T3280 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 757511 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 757511 T3280 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 757512 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 757512 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 757512 T3280 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 757513 T3280 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 757515 T3280 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 757515 T3280 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 757515 T3280 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 757516 T3280 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37831/solr
   [junit4]   2> 757516 T3280 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 757517 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 757518 T3348 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1943e9e name:ZooKeeperConnection Watcher:127.0.0.1:37831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 757519 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 757521 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 757522 T3350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@163655 name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 757522 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 757528 T3280 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 758531 T3280 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54306_zjh
   [junit4]   2> 758532 T3280 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54306_zjh
   [junit4]   2> 758538 T3280 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005
   [junit4]   2> 758544 T3280 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-005/conf/
   [junit4]   2> 758671 T3280 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 758673 T3280 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 758673 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 758921 T3280 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 758922 T3280 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 758925 T3280 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 758926 T3280 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43181
   [junit4]   2> 758927 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 758927 T3280 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 758928 T3280 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006
   [junit4]   2> 758928 T3280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/'
   [junit4]   2> 758941 T3280 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solr.xml
   [junit4]   2> 758970 T3280 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006
   [junit4]   2> 758971 T3280 oasc.CoreContainer.<init> New CoreContainer 19065708
   [junit4]   2> 758971 T3280 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/]
   [junit4]   2> 758972 T3280 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 758972 T3280 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 758973 T3280 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 758973 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 758973 T3280 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 758974 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 758974 T3280 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 758974 T3280 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 758975 T3280 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 758977 T3280 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 758977 T3280 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 758977 T3280 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 758978 T3280 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37831/solr
   [junit4]   2> 758978 T3280 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 758979 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 758980 T3362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@872865 name:ZooKeeperConnection Watcher:127.0.0.1:37831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 758980 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 758982 T3280 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 758983 T3364 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c64317 name:ZooKeeperConnection Watcher:127.0.0.1:37831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 758983 T3280 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 759001 T3280 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 760004 T3280 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43181_zjh
   [junit4]   2> 760006 T3280 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43181_zjh
   [junit4]   2> 760013 T3280 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006
   [junit4]   2> 760018 T3280 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/conf/
   [junit4]   2> 760119 T3280 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 760120 T3280 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 760121 T3280 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 760230 T3318 oasha.CollectionsHandler.handleCreateAction Creating Collection : wt=javabin&router.name=compositeId&action=CREATE&router.field=myOwnField&name=solrj_collection&numShards=2&version=2&replicationFactor=2&collection.configName=conf1
   [junit4]   2> 760231 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760232 T3306 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.name":"compositeId",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 760232 T3306 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.name":"compositeId",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 760233 T3306 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 760234 T3306 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 760237 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760238 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 760238 T3305 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 760239 T3305 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 760240 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760241 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760241 T3364 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760241 T3322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760241 T3304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760241 T3309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760241 T3350 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760337 T3306 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 760337 T3306 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:40582_zjh
   [junit4]   2> 760339 T3306 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:43181_zjh
   [junit4]   2> 760340 T3306 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:43091_zjh
   [junit4]   2> 760346 T3306 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53652_zjh
   [junit4]   2> 760520 T3359 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&shard=shard1&action=CREATE&qt=/admin/cores&name=solrj_collection_shard1_replica2&numShards=2&collection=solrj_collection&version=2&collection.configName=conf1
   [junit4]   2> 760523 T3359 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 760524 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760524 T3359 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 760526 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 760527 T3305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43181/zjh",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43181_zjh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 760528 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760535 T3330 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&shard=shard2&action=CREATE&qt=/admin/cores&name=solrj_collection_shard2_replica2&numShards=2&collection=solrj_collection&version=2&collection.configName=conf1
   [junit4]   2> 760537 T3330 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 760539 T3330 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 760539 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760540 T3305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53652/zjh",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53652_zjh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 760545 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760602 T3300 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&shard=shard1&action=CREATE&qt=/admin/cores&name=solrj_collection_shard1_replica1&numShards=2&collection=solrj_collection&version=2&collection.configName=conf1
   [junit4]   2> 760602 T3316 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&shard=shard2&action=CREATE&qt=/admin/cores&name=solrj_collection_shard2_replica1&numShards=2&collection=solrj_collection&version=2&collection.configName=conf1
   [junit4]   2> 760604 T3300 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 760605 T3316 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 760606 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760606 T3300 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 760606 T3316 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 760607 T3305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40582/zjh",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40582_zjh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 760609 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760609 T3305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43091/zjh",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43091_zjh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 760611 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760713 T3304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760713 T3364 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760713 T3309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760713 T3350 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760713 T3322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760713 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 761525 T3359 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 761526 T3359 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 761527 T3359 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 761527 T3359 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 761528 T3359 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 761529 T3359 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/'
   [junit4]   2> 761539 T3330 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 761540 T3330 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 761541 T3330 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 761541 T3330 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 761542 T3330 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 761543 T3330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/'
   [junit4]   2> 761576 T3330 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 761586 T3359 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 761620 T3316 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 761621 T3316 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 761622 T3316 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 761622 T3316 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 761620 T3300 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 761623 T3316 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 761624 T3316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/'
   [junit4]   2> 761624 T3300 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 761625 T3300 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 761625 T3300 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 761626 T3300 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 761631 T3300 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/solrj_collection_shard1_replica1/'
   [junit4]   2> 761633 T3330 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 761635 T3330 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 761643 T3330 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 761657 T3359 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 761658 T3359 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 761673 T3359 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 761674 T3316 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 761683 T3300 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 761744 T3316 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 761746 T3316 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 761753 T3300 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 761754 T3300 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 761760 T3316 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 761768 T3300 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 761996 T3330 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 762001 T3330 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 762004 T3330 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 762012 T3330 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762015 T3330 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762018 T3330 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762020 T3330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762021 T3330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762021 T3330 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762022 T3330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762022 T3330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762023 T3330 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 762023 T3330 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 762023 T3330 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/data/
   [junit4]   2> 762024 T3330 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b14a45
   [junit4]   2> 762024 T3330 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/data
   [junit4]   2> 762025 T3330 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 762025 T3330 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 762026 T3330 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-004/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 762026 T3330 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=53.5791015625, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=2.9366319556428158, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 762027 T3330 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9d137b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@defab0),segFN=segments_1,generation=1}
   [junit4]   2> 762027 T3330 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 762031 T3330 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 762031 T3330 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 762032 T3330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 762032 T3330 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 762032 T3330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 762032 T3330 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 762033 T3330 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 762033 T3330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 762033 T3330 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 762034 T3330 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 762035 T3330 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 762035 T3330 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 762035 T3330 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 762036 T3330 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 762036 T3330 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 762037 T3330 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 762037 T3330 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 762040 T3330 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 762043 T3330 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 762043 T3330 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 762044 T3330 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=20.5048828125, floorSegmentMB=2.0322265625, forceMergeDeletesPctAllowed=6.767061789379087, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 762044 T3330 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9d137b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@defab0),segFN=segments_1,generation=1}
   [junit4]   2> 762045 T3330 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 762045 T3330 oass.SolrIndexSearcher.<init> Opening Searcher@1cfa31e[solrj_collection_shard2_replica2] main
   [junit4]   2> 762045 T3330 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 762046 T3330 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 762047 T3330 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 762047 T3330 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 762047 T3330 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 762048 T3330 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 762048 T3330 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 762048 T3330 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 762049 T3330 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 762049 T3330 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 762052 T3370 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@1cfa31e[solrj_collection_shard2_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 762054 T3330 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 762054 T3330 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:https://127.0.0.1:53652/zjh collection:solrj_collection shard:shard2
   [junit4]   2> 762055 T3330 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 762062 T3330 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 762063 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762064 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762064 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762064 T3330 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=1.2579674E14ms
   [junit4]   2> 762065 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 762067 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762155 T3359 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 762162 T3359 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 762166 T3359 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 762176 T3359 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762181 T3359 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762184 T3316 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 762185 T3304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 762185 T3322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 762184 T3309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 762185 T3364 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 762185 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 762185 T3350 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 762187 T3359 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762188 T3359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762189 T3359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762189 T3359 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762190 T3359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762191 T3359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762191 T3359 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 762192 T3316 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 762192 T3359 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 762193 T3359 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/data/
   [junit4]   2> 762193 T3359 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b14a45
   [junit4]   2> 762194 T3359 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/data
   [junit4]   2> 762195 T3359 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 762195 T3316 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 762195 T3359 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 762196 T3359 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-006/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 762197 T3359 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=53.5791015625, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=2.9366319556428158, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 762198 T3359 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@90cca1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1bf2f42),segFN=segments_1,generation=1}
   [junit4]   2> 762198 T3359 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 762203 T3359 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 762203 T3359 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 762204 T3359 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 762204 T3359 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 762204 T3359 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 762205 T3359 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 762205 T3359 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 762206 T3359 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 762206 T3359 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 762206 T3316 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762207 T3359 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 762207 T3359 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 762208 T3359 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 762208 T3359 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 762209 T3359 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 762223 T3359 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 762224 T3316 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762224 T3359 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 762225 T3359 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 762227 T3359 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 762228 T3300 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 762228 T3316 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762229 T3316 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762230 T3316 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762230 T3359 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 762230 T3316 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762230 T3359 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 762231 T3316 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762232 T3316 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762231 T3359 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=20.5048828125, floorSegmentMB=2.0322265625, forceMergeDeletesPctAllowed=6.767061789379087, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 762232 T3316 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 762233 T3300 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 762233 T3359 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@90cca1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1bf2f42),segFN=segments_1,generation=1}
   [junit4]   2> 762233 T3359 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 762233 T3316 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 762234 T3359 oass.SolrIndexSearcher.<init> Opening Searcher@4b65e6[solrj_collection_shard1_replica2] main
   [junit4]   2> 762234 T3316 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/data/
   [junit4]   2> 762234 T3359 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 762235 T3316 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b14a45
   [junit4]   2> 762235 T3300 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 762236 T3316 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/data
   [junit4]   2> 762236 T3359 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 762237 T3359 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 762236 T3316 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 762237 T3359 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 762238 T3359 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 762237 T3316 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 762238 T3359 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 762238 T3316 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-003/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 762239 T3359 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 762239 T3316 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=53.5791015625, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=2.9366319556428158, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 762239 T3359 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 762240 T3359 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 762240 T3316 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@b54bfc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1d7299c),segFN=segments_1,generation=1}
   [junit4]   2> 762241 T3316 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 762240 T3359 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 762245 T3372 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@4b65e6[solrj_collection_shard1_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 762245 T3300 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762246 T3316 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 762247 T3316 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 762247 T3316 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 762248 T3316 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 762248 T3359 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 762248 T3359 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:https://127.0.0.1:43181/zjh collection:solrj_collection shard:shard1
   [junit4]   2> 762248 T3300 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 762250 T3359 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 762251 T3316 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 762251 T3316 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 762252 T3316 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 762253 T3316 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 762253 T3316 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 762253 T3300 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762254 T3316 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 762255 T3300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762255 T3300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762255 T3316 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 762256 T3316 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 762256 T3300 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 762257 T3316 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 762258 T3300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 762258 T3300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 762258 T3316 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 762259 T3300 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 762258 T3359 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 762259 T3300 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 762259 T3316 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 762260 T3300 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/solrj_collection_shard1_replica1/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001/tempDir-002/solrj_collection_shard1_replica1/data/
   [junit4]   2> 762261 T3316 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 762261 T3300 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b14a45
   [junit4]   2> 762261 T3304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChang

[...truncated too long message...]

 787231 T3280 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-C611100F7AC1DEA3-001
   [junit4]   2> 39944 T3279 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 10 leaked thread(s).
   [junit4]   2> 788008 T3304 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 788011 T3400 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 788124 T3403 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 788124 T3322 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 788233 T3406 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 788233 T3336 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 788327 T3409 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 788327 T3350 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 788431 T3364 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=ar_EG, timezone=Australia/North
   [junit4]   2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=233695680,total=455090176
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestLMDirichletSimilarityFactory, TestAtomicUpdateErrorCases, TestStressLucene, TestFoldingMultitermQuery, TestManagedStopFilterFactory, TestCoreDiscovery, TestIBSimilarityFactory, OutputWriterTest, OverseerRolesTest, TestDistribDocBasedVersion, TestRTGBase, SpellCheckCollatorTest, HdfsRecoveryZkTest, TestComponentsName, CursorPagingTest, ZkSolrClientTest, SolrIndexConfigTest, TestInfoStreamLogging, TestMiniSolrCloudCluster, EnumFieldTest, SuggesterWFSTTest, SolrCoreCheckLockOnStartupTest, TestDFRSimilarityFactory, DistributedDebugComponentTest, BJQParserTest, TestFastWriter, DistributedExpandComponentTest, TestDefaultSearchFieldResource, XsltUpdateRequestHandlerTest, TestFieldCollectionResource, TestLRUCache, DistribCursorPagingTest, TestOmitPositions, TestRandomMergePolicy, DirectSolrConnectionTest, DistanceFunctionTest, TestSolr4Spatial, TestPhraseSuggestions, FileUtilsTest, TestStressRecovery, SolrPluginUtilsTest, TestSolrJ, RangeFacetTest, TestSolrXMLSerializer, StandardRequestHandlerTest, TermVectorComponentDistributedTest, CurrencyFieldOpenExchangeTest, TestCollationField, TestXIncludeConfig, TestSerializedLuceneMatchVersion, TestCodecSupport, TestCursorMarkWithoutUniqueKey, ZkNodePropsTest, TestFieldResource, PrimUtilsTest, CoreAdminRequestStatusTest, TestHashPartitioner, DocumentAnalysisRequestHandlerTest, TestRangeQuery, TestSweetSpotSimilarityFactory, TestQueryTypes, DirectUpdateHandlerTest, TestUpdate, MinimalSchemaTest, HighlighterConfigTest, AddSchemaFieldsUpdateProcessorFactoryTest, StressHdfsTest, AbstractAnalyticsStatsTest, RemoteQueryErrorTest, CollectionsAPIAsyncDistributedZkTest, BasicDistributedZk2Test, TestZkChroot, DocValuesMultiTest, DistributedQueryComponentOptimizationTest, TestReload, ModifyConfFileTest, TestBinaryResponseWriter, PingRequestHandlerTest, TestPartialUpdateDeduplication, MigrateRouteKeyTest, JSONWriterTest, HdfsUnloadDistributedZkTest, FullSolrCloudDistribCmdsTest, LeaderElectionIntegrationTest, DefaultValueUpdateProcessorTest, TestMultiCoreConfBootstrap, TestSimpleQParserPlugin, TestDynamicFieldCollectionResource, TestSolrQueryParserDefaultOperatorResource, HdfsBasicDistributedZk2Test, OpenExchangeRatesOrgProviderTest, TestCSVLoader, DeleteReplicaTest, HdfsChaosMonkeySafeLeaderTest, InfoHandlerTest, TestCollationFieldDocValues, BasicDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 41.25s, 1 test, 1 error <<< FAILURES!

[...truncated 605 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 400 suites, 1679 tests, 1 error, 37 ignored (16 assumptions)

Total time: 61 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b15 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message