lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_20-ea-b15) - Build # 10604 - Failure!
Date Thu, 19 Jun 2014 20:54:48 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10604/
Java: 32bit/jdk1.8.0_20-ea-b15 -server -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.RollingRestartTest.testDistribSearch

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at __randomizedtesting.SeedInfo.seed([4D653270639751DD:CC83BC6814C831E1]:0)
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.Net.bind(Net.java:428)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
	at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:631)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.eclipse.jetty.server.Server.doStart(Server.java:291)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:432)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:405)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:102)
	at org.apache.solr.cloud.RollingRestartTest.doTest(RollingRestartTest.java:61)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10975 lines...]
   [junit4] Suite: org.apache.solr.cloud.RollingRestartTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/init-core-data-001
   [junit4]   2> 92416 T470 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 92416 T470 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /vw_/
   [junit4]   2> 92419 T470 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 92420 T470 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 92420 T471 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 92520 T470 oasc.ZkTestServer.run start zk server on port:40050
   [junit4]   2> 92521 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 92565 T477 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6da3e6 name:ZooKeeperConnection Watcher:127.0.0.1:40050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 92565 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 92566 T470 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 92573 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 92579 T479 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131d425 name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 92579 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 92580 T470 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 92582 T470 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 92586 T470 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 92587 T470 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 92589 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 92590 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 92593 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 92594 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 92597 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 92598 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 92600 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 92601 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 92603 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 92604 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 92606 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 92606 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 92608 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 92609 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 92612 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 92612 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 92615 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 92615 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 92618 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 92618 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 92620 T470 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 92621 T470 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 92627 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 92628 T481 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15a5864 name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 92629 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93003 T470 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 93007 T470 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 93009 T470 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49934
   [junit4]   2> 93011 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 93012 T470 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 93012 T470 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002
   [junit4]   2> 93013 T470 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/'
   [junit4]   2> 93023 T470 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/solr.xml
   [junit4]   2> 93037 T470 oasc.CoreContainer.<init> New CoreContainer 18118299
   [junit4]   2> 93038 T470 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/]
   [junit4]   2> 93038 T470 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 93038 T470 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 93039 T470 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 93039 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 93039 T470 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 93040 T470 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 93040 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 93040 T470 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 93040 T470 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 93042 T470 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 93043 T470 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 93043 T470 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 93043 T470 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40050/solr
   [junit4]   2> 93044 T470 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 93044 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 93047 T492 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64e5d1 name:ZooKeeperConnection Watcher:127.0.0.1:40050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 93048 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93049 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 93051 T494 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a5ad03 name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 93051 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93052 T470 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 93054 T470 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 93056 T470 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 93058 T470 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 93060 T470 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 93063 T470 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 93067 T470 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49934_vw_
   [junit4]   2> 93070 T470 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49934_vw_
   [junit4]   2> 93072 T470 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 93073 T470 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 93075 T470 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49934_vw_
   [junit4]   2> 93076 T470 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 93077 T470 oasc.Overseer.start Overseer (id=91960759187800068-127.0.0.1:49934_vw_-n_0000000000) starting
   [junit4]   2> 93079 T470 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 93088 T496 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 93088 T470 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 93090 T470 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 93091 T470 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 93093 T495 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 93095 T497 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 93095 T497 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 93097 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93097 T497 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 93098 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 93098 T495 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49934/vw_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49934_vw_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 93099 T495 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 93099 T495 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 93100 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93101 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 94097 T497 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 94098 T497 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 94098 T497 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 94098 T497 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 94099 T497 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 94099 T497 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/collection1/'
   [junit4]   2> 94100 T497 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 94100 T497 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 94100 T497 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 94117 T497 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 94131 T497 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 94132 T497 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 94136 T497 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 94230 T497 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 94234 T497 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 94235 T497 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 94242 T497 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 94244 T497 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 94246 T497 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 94247 T497 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 94247 T497 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 94247 T497 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 94248 T497 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 94248 T497 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 94248 T497 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 94249 T497 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 94256 T497 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data/
   [junit4]   2> 94256 T497 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e609b7
   [junit4]   2> 94256 T497 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data
   [junit4]   2> 94257 T497 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data/index/
   [junit4]   2> 94257 T497 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 94257 T497 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data/index
   [junit4]   2> 94258 T497 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=57.5390625, floorSegmentMB=0.796875, forceMergeDeletesPctAllowed=12.282954279801386, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 94262 T497 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 94262 T497 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 94264 T497 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 94265 T497 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 94265 T497 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 94265 T497 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 94265 T497 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 94265 T497 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 94266 T497 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 94266 T497 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 94266 T497 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 94266 T497 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 94267 T497 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 94267 T497 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 94267 T497 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 94267 T497 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 94267 T497 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 94268 T497 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 94268 T497 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 94269 T497 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 94270 T497 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 94271 T497 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 94271 T497 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 94272 T497 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 94272 T497 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 94272 T497 oass.SolrIndexSearcher.<init> Opening Searcher@c3e7ba[collection1] main
   [junit4]   2> 94273 T497 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 94273 T497 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 94274 T497 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 94274 T497 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 94274 T497 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 94274 T497 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 94275 T497 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 94275 T497 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 94275 T497 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 94276 T497 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 94280 T497 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 94280 T501 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49934/vw_ collection:control_collection shard:shard1
   [junit4]   2> 94281 T470 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 94281 T501 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 94281 T498 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c3e7ba[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 94281 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 94285 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94287 T503 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19d53f9 name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94287 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94288 T470 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 94289 T501 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 94290 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94290 T501 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 94291 T501 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C277 name=collection1 org.apache.solr.core.SolrCore@d508eb url=https://127.0.0.1:49934/vw_/collection1 node=127.0.0.1:49934_vw_ C277_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:49934/vw_, core=collection1, node_name=127.0.0.1:49934_vw_}
   [junit4]   2> 94291 T501 C277 P49934 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49934/vw_/collection1/
   [junit4]   2> 94291 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 94291 T501 C277 P49934 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 94291 T470 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 94292 T501 C277 P49934 oasc.SyncStrategy.syncToMe https://127.0.0.1:49934/vw_/collection1/ has no replicas
   [junit4]   2> 94292 T501 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49934/vw_/collection1/ shard1
   [junit4]   2> 94292 T501 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 94294 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94298 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94301 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94402 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 94402 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 94449 T501 oasc.ZkController.register We are https://127.0.0.1:49934/vw_/collection1/ and leader is https://127.0.0.1:49934/vw_/collection1/
   [junit4]   2> 94450 T501 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49934/vw_
   [junit4]   2> 94450 T501 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 94450 T501 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 94450 T501 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 94452 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94452 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94452 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94452 T501 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 94453 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 94454 T495 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49934/vw_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49934_vw_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 94455 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 94543 T470 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 94544 T470 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 94549 T470 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 94551 T470 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:47649
   [junit4]   2> 94553 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 94554 T470 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 94554 T470 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003
   [junit4]   2> 94554 T470 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/'
   [junit4]   2> 94565 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 94565 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 94574 T470 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/solr.xml
   [junit4]   2> 94581 T470 oasc.CoreContainer.<init> New CoreContainer 30817219
   [junit4]   2> 94582 T470 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/]
   [junit4]   2> 94582 T470 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 94583 T470 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 94583 T470 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 94583 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 94583 T470 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 94584 T470 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 94584 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 94584 T470 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 94584 T470 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 94586 T470 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 94587 T470 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 94587 T470 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 94587 T470 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40050/solr
   [junit4]   2> 94588 T470 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 94588 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94590 T514 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dc7b2 name:ZooKeeperConnection Watcher:127.0.0.1:40050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94590 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94592 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94593 T516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32d4df name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94593 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94597 T470 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 95600 T470 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47649_vw_
   [junit4]   2> 95602 T470 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47649_vw_
   [junit4]   2> 95611 T517 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 95611 T517 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 95612 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95612 T517 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 95612 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95612 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95613 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 95614 T495 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47649/vw_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47649_vw_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 95614 T495 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 95614 T495 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 95615 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95717 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 95717 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 95717 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 96612 T517 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 96613 T517 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 96613 T517 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 96614 T517 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 96614 T517 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 96615 T517 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/collection1/'
   [junit4]   2> 96615 T517 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 96615 T517 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 96616 T517 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 96639 T517 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 96648 T517 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 96649 T517 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 96653 T517 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 96750 T517 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 96753 T517 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 96755 T517 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 96762 T517 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 96764 T517 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 96766 T517 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 96767 T517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 96767 T517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 96767 T517 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 96768 T517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 96768 T517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 96768 T517 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 96768 T517 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 96768 T517 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1/
   [junit4]   2> 96769 T517 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e609b7
   [junit4]   2> 96769 T517 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1
   [junit4]   2> 96769 T517 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1/index/
   [junit4]   2> 96770 T517 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 96770 T517 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1/index
   [junit4]   2> 96770 T517 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=57.5390625, floorSegmentMB=0.796875, forceMergeDeletesPctAllowed=12.282954279801386, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 96776 T517 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 96776 T517 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 96779 T517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 96779 T517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 96779 T517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 96780 T517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 96780 T517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 96780 T517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 96780 T517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 96781 T517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 96781 T517 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 96781 T517 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 96781 T517 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 96782 T517 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 96782 T517 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 96782 T517 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 96782 T517 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 96783 T517 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 96783 T517 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 96784 T517 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 96785 T517 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 96786 T517 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 96786 T517 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 96787 T517 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 96787 T517 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 96788 T517 oass.SolrIndexSearcher.<init> Opening Searcher@18d3c80[collection1] main
   [junit4]   2> 96788 T517 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 96789 T517 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 96789 T517 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 96789 T517 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 96790 T517 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 96790 T517 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 96790 T517 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 96790 T517 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 96791 T517 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 96791 T517 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 96795 T518 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18d3c80[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 96796 T517 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 96797 T521 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:47649/vw_ collection:collection1 shard:shard2
   [junit4]   2> 96797 T470 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 96798 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 96804 T521 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 96814 T521 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 96817 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96817 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96817 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96818 T521 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 96818 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 96818 T521 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C278 name=collection1 org.apache.solr.core.SolrCore@198bad6 url=https://127.0.0.1:47649/vw_/collection1 node=127.0.0.1:47649_vw_ C278_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:47649/vw_, core=collection1, node_name=127.0.0.1:47649_vw_}
   [junit4]   2> 96818 T521 C278 P47649 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:47649/vw_/collection1/
   [junit4]   2> 96819 T521 C278 P47649 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 96819 T521 C278 P47649 oasc.SyncStrategy.syncToMe https://127.0.0.1:47649/vw_/collection1/ has no replicas
   [junit4]   2> 96819 T521 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:47649/vw_/collection1/ shard2
   [junit4]   2> 96819 T521 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 96820 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96823 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96827 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96928 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 96928 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 96928 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 96976 T521 oasc.ZkController.register We are https://127.0.0.1:47649/vw_/collection1/ and leader is https://127.0.0.1:47649/vw_/collection1/
   [junit4]   2> 96977 T521 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:47649/vw_
   [junit4]   2> 96977 T521 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 96977 T521 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 96977 T521 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 96978 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96978 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96978 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 96979 T521 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 96979 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 96980 T495 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47649/vw_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47649_vw_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 96981 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 97054 T470 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 97055 T470 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 97059 T470 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 97060 T470 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35530
   [junit4]   2> 97062 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 97062 T470 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 97062 T470 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004
   [junit4]   2> 97063 T470 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/'
   [junit4]   2> 97073 T470 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/solr.xml
   [junit4]   2> 97087 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 97087 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 97087 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 97087 T470 oasc.CoreContainer.<init> New CoreContainer 17425716
   [junit4]   2> 97088 T470 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/]
   [junit4]   2> 97089 T470 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 97089 T470 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 97089 T470 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 97090 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 97090 T470 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 97090 T470 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 97090 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 97091 T470 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 97091 T470 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 97093 T470 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 97094 T470 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 97094 T470 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 97094 T470 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40050/solr
   [junit4]   2> 97095 T470 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 97095 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97097 T532 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12ea145 name:ZooKeeperConnection Watcher:127.0.0.1:40050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 97097 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 97099 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97100 T534 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13bda03 name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 97100 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 97105 T470 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 98106 T470 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35530_vw_
   [junit4]   2> 98111 T470 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35530_vw_
   [junit4]   2> 98117 T535 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 98117 T535 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 98118 T535 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 98118 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 98118 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 98119 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 98119 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 98120 T495 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35530/vw_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35530_vw_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 98120 T495 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 98120 T495 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 98122 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 98223 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 98223 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 98223 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 98224 T534 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99118 T535 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 99119 T535 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 99119 T535 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 99120 T535 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 99121 T535 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 99121 T535 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/collection1/'
   [junit4]   2> 99121 T535 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 99121 T535 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 99122 T535 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 99141 T535 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 99155 T535 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 99156 T535 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 99160 T535 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 99253 T535 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 99256 T535 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 99258 T535 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 99271 T535 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 99273 T535 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 99275 T535 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 99276 T535 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 99276 T535 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 99276 T535 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 99277 T535 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 99277 T535 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 99277 T535 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 99278 T535 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 99278 T535 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2/
   [junit4]   2> 99278 T535 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e609b7
   [junit4]   2> 99278 T535 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2
   [junit4]   2> 99279 T535 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2/index/
   [junit4]   2> 99279 T535 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 99279 T535 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2/index
   [junit4]   2> 99280 T535 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=57.5390625, floorSegmentMB=0.796875, forceMergeDeletesPctAllowed=12.282954279801386, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 99286 T535 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 99287 T535 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 99290 T535 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 99290 T535 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 99291 T535 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 99291 T535 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 99291 T535 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 99291 T535 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 99291 T535 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 99292 T535 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 99292 T535 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 99292 T535 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 99293 T535 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 99293 T535 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 99293 T535 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 99293 T535 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 99293 T535 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 99294 T535 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 99294 T535 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 99296 T535 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 99297 T535 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 99297 T535 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 99297 T535 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 99298 T535 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 99299 T535 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 99299 T535 oass.SolrIndexSearcher.<init> Opening Searcher@9c7cfc[collection1] main
   [junit4]   2> 99299 T535 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 99300 T535 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 99300 T535 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 99301 T535 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 99301 T535 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 99301 T535 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 99302 T535 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 99302 T535 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 99302 T535 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 99302 T535 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 99305 T536 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9c7cfc[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 99308 T535 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 99309 T539 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35530/vw_ collection:collection1 shard:shard1
   [junit4]   2> 99309 T470 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 99310 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 99312 T539 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 99318 T539 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 99320 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99320 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99320 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99320 T539 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 99321 T539 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 99321 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C279 name=collection1 org.apache.solr.core.SolrCore@1df9394 url=https://127.0.0.1:35530/vw_/collection1 node=127.0.0.1:35530_vw_ C279_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:35530/vw_, core=collection1, node_name=127.0.0.1:35530_vw_}
   [junit4]   2> 99321 T539 C279 P35530 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35530/vw_/collection1/
   [junit4]   2> 99321 T539 C279 P35530 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 99322 T539 C279 P35530 oasc.SyncStrategy.syncToMe https://127.0.0.1:35530/vw_/collection1/ has no replicas
   [junit4]   2> 99322 T539 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35530/vw_/collection1/ shard1
   [junit4]   2> 99322 T539 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 99324 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99330 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99333 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99434 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99434 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99434 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99434 T534 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99482 T539 oasc.ZkController.register We are https://127.0.0.1:35530/vw_/collection1/ and leader is https://127.0.0.1:35530/vw_/collection1/
   [junit4]   2> 99482 T539 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35530/vw_
   [junit4]   2> 99482 T539 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 99482 T539 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 99482 T539 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 99484 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99485 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99485 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99485 T539 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 99486 T495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 99486 T495 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35530/vw_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35530_vw_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 99488 T494 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 99561 T470 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 99562 T470 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 99567 T470 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 99569 T470 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45384
   [junit4]   2> 99572 T470 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 99573 T470 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 99573 T470 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-005
   [junit4]   2> 99573 T470 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-005/'
   [junit4]   2> 99586 T470 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-005/solr.xml
   [junit4]   2> 99597 T534 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99597 T503 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99597 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99598 T494 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 99602 T470 oasc.CoreContainer.<init> New CoreContainer 24114730
   [junit4]   2> 99602 T470 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-005/]
   [junit4]   2> 99603 T470 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 99603 T470 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 99603 T470 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 99604 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 99604 T470 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 99604 T470 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 99604 T470 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 99605 T470 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 99605 T470 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 99607 T470 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 99607 T470 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 99607 T470 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 99608 T470 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40050/solr
   [junit4]   2> 99608 T470 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 99609 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 99610 T550 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18fac95 name:ZooKeeperConnection Watcher:127.0.0.1:40050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 99610 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 99612 T470 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 99613 T552 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f3fc79 name:ZooKeeperConnection Watcher:127.0.0.1:40050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 99613 T470 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4] 

[...truncated too long message...]

ed, but Solr cannot talk to ZK
   [junit4]   2> ASYNC  NEW_CORE C359 name=collection1 org.apache.solr.core.SolrCore@a43f58 url=https://127.0.0.1:37901/vw_/collection1 node=127.0.0.1:37901_vw_ C359_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:37901/vw_, core=collection1, node_name=127.0.0.1:37901_vw_}
   [junit4]   2> 192495 T824 C359 P37901 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 192496 T824 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 192496 T824 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:37901/vw_/collection1/ shard1
   [junit4]   2> 192496 T824 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@a43f58
   [junit4]   2> 192496 T824 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 192497 T824 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 192497 T824 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 192497 T824 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 192498 T824 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 192498 T824 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 192498 T824 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty16/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty16/index;done=false>>]
   [junit4]   2> 192499 T824 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty16/index
   [junit4]   2> 192499 T824 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty16 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty16;done=false>>]
   [junit4]   2> 192499 T824 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-4D653270639751DD-001/tempDir-001/jetty16
   [junit4]   2> 192499 T824 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 192500 T824 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:139)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:294)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:357)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 	Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:209)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:417)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:131)
   [junit4]   2> 		at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:126)
   [junit4]   2> 		... 7 more
   [junit4]   2> 	
   [junit4]   2> 192500 T824 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 192500 T824 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:357)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-4D653270639751DD-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ca, timezone=Asia/Ulan_Bator
   [junit4]   2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.8.0_20-ea (32-bit)/cpus=8,threads=1,free=13421560,total=124538880
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastLRUCache, TestCopyFieldCollectionResource, AutoCommitTest, CircularListTest, DocValuesTest, TestSearchPerf, OverseerCollectionProcessorTest, CSVRequestHandlerTest, TestShardHandlerFactory, FileBasedSpellCheckerTest, TestJmxMonitoredMap, TestFaceting, TestValueSourceCache, CursorPagingTest, SpatialFilterTest, SliceStateUpdateTest, TestMiniSolrCloudCluster, ClusterStateUpdateTest, SOLR749Test, TestConfigSets, BasicFunctionalityTest, StatelessScriptUpdateProcessorFactoryTest, TestAnalyzeInfixSuggestions, TestFieldCollectionResource, SchemaVersionSpecificBehaviorTest, SliceStateTest, TestBadConfig, RollingRestartTest]
   [junit4] Completed on J0 in 100.70s, 1 test, 1 error <<< FAILURES!

[...truncated 1143 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 415 suites, 1703 tests, 1 error, 36 ignored (15 assumptions)

Total time: 67 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_20-ea-b15 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message