lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 2021 - Still Failing
Date Fri, 04 Jul 2014 01:59:00 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/2021/

3 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([84A7796AF29B22FB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([84A7796AF29B22FB]:0)


REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1
	at __randomizedtesting.SeedInfo.seed([84A7796AF29B22FB:541F77285C442C7]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:545)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:350)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:149)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11188 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/init-core-data-001
   [junit4]   2> 2815921 T4107 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/
   [junit4]   2> 2815927 T4107 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2815928 T4107 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2815929 T4108 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2816030 T4107 oasc.ZkTestServer.run start zk server on port:52399
   [junit4]   2> 2816032 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2816036 T4114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74c6ed97 name:ZooKeeperConnection Watcher:127.0.0.1:52399 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2816036 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2816036 T4107 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2816047 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2816049 T4116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25883166 name:ZooKeeperConnection Watcher:127.0.0.1:52399/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2816049 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2816049 T4107 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2816059 T4107 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2816062 T4107 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2816064 T4107 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2816073 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2816074 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2816086 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2816087 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2816192 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2816193 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2816196 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2816196 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2816199 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2816200 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2816207 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2816208 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2816211 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2816211 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2816214 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2816215 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2816223 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2816223 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2816226 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2816227 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2816230 T4107 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2816230 T4107 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2817318 T4107 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2817323 T4107 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:52403
   [junit4]   2> 2817323 T4107 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2817323 T4107 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2817324 T4107 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002
   [junit4]   2> 2817324 T4107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/'
   [junit4]   2> 2817357 T4107 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/solr.xml
   [junit4]   2> 2817438 T4107 oasc.CoreContainer.<init> New CoreContainer 1848427469
   [junit4]   2> 2817438 T4107 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/]
   [junit4]   2> 2817439 T4107 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2817439 T4107 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2817439 T4107 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2817440 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2817440 T4107 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2817440 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2817440 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2817441 T4107 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2817441 T4107 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2817441 T4107 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2817442 T4107 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2817442 T4107 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2817442 T4107 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52399/solr
   [junit4]   2> 2817443 T4107 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2817443 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2817446 T4127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ceb003d name:ZooKeeperConnection Watcher:127.0.0.1:52399 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2817447 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2817461 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2817463 T4129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77d4e3c0 name:ZooKeeperConnection Watcher:127.0.0.1:52399/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2817463 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2817465 T4107 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2817469 T4107 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2817472 T4107 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2817475 T4107 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2817478 T4107 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2817481 T4107 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2817483 T4107 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52402__
   [junit4]   2> 2817484 T4107 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52402__
   [junit4]   2> 2817487 T4107 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2817489 T4107 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2817500 T4107 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52402__
   [junit4]   2> 2817501 T4107 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2817503 T4107 oasc.Overseer.start Overseer (id=92040992474202115-127.0.0.1:52402__-n_0000000000) starting
   [junit4]   2> 2817513 T4107 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2817538 T4131 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2817538 T4107 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2817548 T4107 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2817549 T4107 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2817551 T4130 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2817555 T4132 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2817555 T4132 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2817556 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2817556 T4132 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2817557 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2817558 T4130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52402/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52402__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2817558 T4130 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2817558 T4130 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2817561 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2817570 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2818558 T4132 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2818559 T4132 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2818560 T4132 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2818560 T4132 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2818561 T4132 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2818561 T4132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/collection1/'
   [junit4]   2> 2818562 T4132 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2818563 T4132 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2818563 T4132 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2818630 T4132 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 2818678 T4132 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2818780 T4132 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2818793 T4132 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2819222 T4132 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2819223 T4132 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2819224 T4132 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2819237 T4132 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2819240 T4132 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2819252 T4132 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2819257 T4132 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2819261 T4132 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2819261 T4132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2819262 T4132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2819262 T4132 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2819263 T4132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2819263 T4132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2819264 T4132 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2819264 T4132 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2819264 T4132 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/control/data/
   [junit4]   2> 2819265 T4132 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@299a3d4
   [junit4]   2> 2819265 T4132 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/control/data
   [junit4]   2> 2819266 T4132 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/control/data/index/
   [junit4]   2> 2819266 T4132 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2819266 T4132 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/control/data/index
   [junit4]   2> 2819267 T4132 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=84.078125, floorSegmentMB=0.9296875, forceMergeDeletesPctAllowed=17.506918862156883, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2819268 T4132 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6fe6e00e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@37cd0f19),segFN=segments_1,generation=1}
   [junit4]   2> 2819268 T4132 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2819272 T4132 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2819273 T4132 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2819273 T4132 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2819273 T4132 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2819274 T4132 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2819274 T4132 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2819275 T4132 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2819275 T4132 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2819275 T4132 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2819276 T4132 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2819276 T4132 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2819277 T4132 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2819277 T4132 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2819277 T4132 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2819278 T4132 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2819278 T4132 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2819297 T4132 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2819303 T4132 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2819303 T4132 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2819304 T4132 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=98.578125, floorSegmentMB=1.853515625, forceMergeDeletesPctAllowed=9.9755902801654, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2819304 T4132 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6fe6e00e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@37cd0f19),segFN=segments_1,generation=1}
   [junit4]   2> 2819305 T4132 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2819305 T4132 oass.SolrIndexSearcher.<init> Opening Searcher@227d51bc[collection1] main
   [junit4]   2> 2819305 T4132 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2819306 T4132 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2819306 T4132 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2819307 T4132 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2819307 T4132 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2819307 T4132 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2819308 T4132 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2819308 T4132 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2819309 T4132 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2819309 T4132 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2819312 T4133 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@227d51bc[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2819314 T4132 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2819315 T4136 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52402/_ collection:control_collection shard:shard1
   [junit4]   2> 2819315 T4107 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2819316 T4107 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2819316 T4136 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2819317 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2819320 T4139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@523cc68c name:ZooKeeperConnection Watcher:127.0.0.1:52399/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2819321 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2819334 T4107 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2819346 T4136 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2819346 T4107 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2819348 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819348 T4136 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2819349 T4136 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C712 name=collection1 org.apache.solr.core.SolrCore@313216db url=http://127.0.0.1:52402/_/collection1 node=127.0.0.1:52402__ C712_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:52402/_, core=collection1, node_name=127.0.0.1:52402__}
   [junit4]   2> 2819349 T4136 C712 P52402 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52402/_/collection1/
   [junit4]   2> 2819349 T4136 C712 P52402 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2819349 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2819349 T4136 C712 P52402 oasc.SyncStrategy.syncToMe http://127.0.0.1:52402/_/collection1/ has no replicas
   [junit4]   2> 2819350 T4136 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52402/_/collection1/ shard1
   [junit4]   2> 2819350 T4136 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2819359 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819367 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2819367 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2819380 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2819383 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819486 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2819486 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2819532 T4136 oasc.ZkController.register We are http://127.0.0.1:52402/_/collection1/ and leader is http://127.0.0.1:52402/_/collection1/
   [junit4]   2> 2819533 T4136 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52402/_
   [junit4]   2> 2819533 T4136 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2819533 T4136 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2819533 T4136 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2819535 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819535 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819535 T4136 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2819535 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819537 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2819537 T4130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52402/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52402__",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2819540 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2819643 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2819643 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2820388 T4107 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2820389 T4107 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2820391 T4107 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:52408
   [junit4]   2> 2820392 T4107 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2820392 T4107 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2820392 T4107 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003
   [junit4]   2> 2820393 T4107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/'
   [junit4]   2> 2820426 T4107 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/solr.xml
   [junit4]   2> 2820501 T4107 oasc.CoreContainer.<init> New CoreContainer 880926921
   [junit4]   2> 2820502 T4107 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/]
   [junit4]   2> 2820503 T4107 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2820503 T4107 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2820503 T4107 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2820503 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2820503 T4107 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2820504 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2820504 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2820504 T4107 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2820504 T4107 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2820505 T4107 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2820505 T4107 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2820506 T4107 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2820506 T4107 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52399/solr
   [junit4]   2> 2820506 T4107 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2820507 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2820533 T4150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3accc70d name:ZooKeeperConnection Watcher:127.0.0.1:52399 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2820533 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2820549 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2820551 T4152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@736bc79b name:ZooKeeperConnection Watcher:127.0.0.1:52399/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2820551 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2820576 T4107 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2821579 T4107 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52407__
   [junit4]   2> 2821581 T4107 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52407__
   [junit4]   2> 2821585 T4139 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2821585 T4152 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2821585 T4129 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2821600 T4153 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2821600 T4153 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2821601 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2821601 T4153 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2821602 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2821602 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2821603 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2821603 T4130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52407/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52407__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2821604 T4130 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2821604 T4130 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2821607 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2821617 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2821617 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2821617 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2822603 T4153 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2822604 T4153 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2822604 T4153 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2822605 T4153 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2822606 T4153 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2822606 T4153 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/collection1/'
   [junit4]   2> 2822607 T4153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2822608 T4153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2822608 T4153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2822675 T4153 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 2822723 T4153 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2822824 T4153 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2822835 T4153 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2823262 T4153 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2823264 T4153 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2823264 T4153 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2823277 T4153 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2823280 T4153 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2823292 T4153 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2823296 T4153 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2823300 T4153 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2823301 T4153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2823302 T4153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2823302 T4153 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2823303 T4153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2823303 T4153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2823303 T4153 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2823304 T4153 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2823304 T4153 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty1/
   [junit4]   2> 2823304 T4153 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@299a3d4
   [junit4]   2> 2823305 T4153 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty1
   [junit4]   2> 2823305 T4153 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty1/index/
   [junit4]   2> 2823306 T4153 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2823306 T4153 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty1/index
   [junit4]   2> 2823306 T4153 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=84.078125, floorSegmentMB=0.9296875, forceMergeDeletesPctAllowed=17.506918862156883, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2823307 T4153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@46875b26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@576e8454),segFN=segments_1,generation=1}
   [junit4]   2> 2823308 T4153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2823313 T4153 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2823314 T4153 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2823314 T4153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2823314 T4153 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2823315 T4153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2823315 T4153 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2823316 T4153 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2823316 T4153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2823316 T4153 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2823317 T4153 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2823317 T4153 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2823318 T4153 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2823318 T4153 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2823319 T4153 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2823319 T4153 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2823320 T4153 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2823339 T4153 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2823343 T4153 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2823343 T4153 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2823344 T4153 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=98.578125, floorSegmentMB=1.853515625, forceMergeDeletesPctAllowed=9.9755902801654, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2823345 T4153 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@46875b26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@576e8454),segFN=segments_1,generation=1}
   [junit4]   2> 2823345 T4153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2823345 T4153 oass.SolrIndexSearcher.<init> Opening Searcher@214c6b9a[collection1] main
   [junit4]   2> 2823346 T4153 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2823347 T4153 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2823347 T4153 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2823347 T4153 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2823348 T4153 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2823348 T4153 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2823348 T4153 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2823349 T4153 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2823349 T4153 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2823349 T4153 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2823354 T4154 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@214c6b9a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2823356 T4153 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2823357 T4157 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52407/_ collection:collection1 shard:shard1
   [junit4]   2> 2823358 T4107 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2823358 T4107 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2823358 T4157 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2823375 T4157 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2823378 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823378 T4157 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2823378 T4157 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C713 name=collection1 org.apache.solr.core.SolrCore@471bc16a url=http://127.0.0.1:52407/_/collection1 node=127.0.0.1:52407__ C713_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52407/_, core=collection1, node_name=127.0.0.1:52407__}
   [junit4]   2> 2823378 T4157 C713 P52407 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52407/_/collection1/
   [junit4]   2> 2823379 T4157 C713 P52407 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2823379 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2823379 T4157 C713 P52407 oasc.SyncStrategy.syncToMe http://127.0.0.1:52407/_/collection1/ has no replicas
   [junit4]   2> 2823379 T4157 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52407/_/collection1/ shard1
   [junit4]   2> 2823380 T4157 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2823382 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823383 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823384 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823383 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823406 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2823409 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823512 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823512 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823512 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823559 T4157 oasc.ZkController.register We are http://127.0.0.1:52407/_/collection1/ and leader is http://127.0.0.1:52407/_/collection1/
   [junit4]   2> 2823560 T4157 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52407/_
   [junit4]   2> 2823560 T4157 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2823560 T4157 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2823560 T4157 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2823562 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823562 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823562 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823562 T4157 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2823563 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2823564 T4130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52407/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52407__",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2823573 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2823676 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823676 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2823676 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2824394 T4107 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2824395 T4107 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2824397 T4107 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:52412
   [junit4]   2> 2824398 T4107 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2824398 T4107 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2824398 T4107 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004
   [junit4]   2> 2824399 T4107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/'
   [junit4]   2> 2824432 T4107 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/solr.xml
   [junit4]   2> 2824507 T4107 oasc.CoreContainer.<init> New CoreContainer 2119335915
   [junit4]   2> 2824508 T4107 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/]
   [junit4]   2> 2824509 T4107 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2824509 T4107 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2824509 T4107 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2824509 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2824510 T4107 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2824510 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2824510 T4107 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2824510 T4107 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2824511 T4107 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2824511 T4107 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2824511 T4107 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2824512 T4107 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2824512 T4107 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52399/solr
   [junit4]   2> 2824512 T4107 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2824513 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2824527 T4169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a4c5e17 name:ZooKeeperConnection Watcher:127.0.0.1:52399 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2824527 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2824542 T4107 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2824544 T4171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e6eb68e name:ZooKeeperConnection Watcher:127.0.0.1:52399/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2824544 T4107 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2824569 T4107 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2825573 T4107 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52411__
   [junit4]   2> 2825575 T4107 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52411__
   [junit4]   2> 2825579 T4152 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2825579 T4171 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2825579 T4139 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2825579 T4129 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2825597 T4172 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2825597 T4172 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2825599 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2825599 T4172 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2825599 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2825599 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2825600 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2825600 T4130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52411/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52411__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2825601 T4130 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2825601 T4130 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2825604 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2825605 T4171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2825605 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2825605 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2825605 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2826600 T4172 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2826601 T4172 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2826601 T4172 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2826602 T4172 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2826602 T4172 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2826603 T4172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/collection1/'
   [junit4]   2> 2826604 T4172 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2826605 T4172 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2826605 T4172 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2826671 T4172 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 2826719 T4172 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2826821 T4172 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2826832 T4172 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2827258 T4172 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2827259 T4172 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2827260 T4172 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2827273 T4172 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2827276 T4172 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2827288 T4172 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2827292 T4172 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2827296 T4172 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2827297 T4172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2827297 T4172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2827298 T4172 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2827299 T4172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2827299 T4172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2827299 T4172 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2827299 T4172 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2827300 T4172 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty2/
   [junit4]   2> 2827300 T4172 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@299a3d4
   [junit4]   2> 2827301 T4172 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty2
   [junit4]   2> 2827301 T4172 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty2/index/
   [junit4]   2> 2827301 T4172 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2827302 T4172 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-84A7796AF29B22FB-001/tempDir-001/jetty2/index
   [junit4]   2> 2827302 T4172 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=84.078125, floorSegmentMB=0.9296875, forceMergeDeletesPctAllowed=17.506918862156883, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2827303 T4172 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@71528b6e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@45e89ddb),segFN=segments_1,generation=1}
   [junit4]   2> 2827303 T4172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2827309 T4172 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2827310 T4172 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2827310 T4172 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2827310 T4172 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2827310 T4172 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2827311 T4172 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2827311 T4172 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2827312 T4172 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2827312 T4172 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2827313 T4172 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2827313 T4172 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2827313 T4172 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2827314 T4172 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2827314 T4172 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2827315 T4172 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2827316 T4172 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2827334 T4172 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2827339 T4172 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2827339 T4172 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2827340 T4172 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=98.578125, floorSegmentMB=1.853515625, forceMergeDeletesPctAllowed=9.9755902801654, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2827340 T4172 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@71528b6e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@45e89ddb),segFN=segments_1,generation=1}
   [junit4]   2> 2827341 T4172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2827341 T4172 oass.SolrIndexSearcher.<init> Opening Searcher@18a4dc0b[collection1] main
   [junit4]   2> 2827341 T4172 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2827342 T4172 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2827342 T4172 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2827343 T4172 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2827343 T4172 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2827343 T4172 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2827344 T4172 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2827344 T4172 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2827345 T4172 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2827345 T4172 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2827349 T4173 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18a4dc0b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2827352 T4172 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2827352 T4176 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52411/_ collection:collection1 shard:shard2
   [junit4]   2> 2827353 T4107 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2827353 T4107 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2827353 T4176 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2827355 T4107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2827356 T4107 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 2827356 T4107 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 2827357 T4107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2827370 T4176 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2827379 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827380 T4176 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2827380 T4176 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C714 name=collection1 org.apache.solr.core.SolrCore@5413c6d8 url=http://127.0.0.1:52411/_/collection1 node=127.0.0.1:52411__ C714_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52411/_, core=collection1, node_name=127.0.0.1:52411__}
   [junit4]   2> 2827380 T4176 C714 P52411 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52411/_/collection1/
   [junit4]   2> 2827380 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2827380 T4176 C714 P52411 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2827381 T4176 C714 P52411 oasc.SyncStrategy.syncToMe http://127.0.0.1:52411/_/collection1/ has no replicas
   [junit4]   2> 2827381 T4176 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52411/_/collection1/ shard2
   [junit4]   2> 2827381 T4176 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2827383 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827384 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827384 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827384 T4171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827384 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827388 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2827391 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827494 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827494 T4171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827494 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827494 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827540 T4176 oasc.ZkController.register We are http://127.0.0.1:52411/_/collection1/ and leader is http://127.0.0.1:52411/_/collection1/
   [junit4]   2> 2827541 T4176 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52411/_
   [junit4]   2> 2827541 T4176 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2827541 T4176 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2827541 T4176 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2827543 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827543 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827543 T4176 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2827543 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827544 T4130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2827545 T4130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52411/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52411__",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2827554 T4129 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2827657 T4129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827657 T4171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827657 T4152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2827657 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2828360 T4107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2828361 T4107 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
   [junit4]   2> 2828362 T4137 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=52415,localport=52402], receiveBufferSize:6565888
   [junit4]   2> 2828363 T4137 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=52403,localport=52416], receiveBufferSize=6565888
   [junit4]   2> ASYNC  NEW_CORE C715 name=collection1 org.apache.solr.core.SolrCore@313216db url=http://127.0.0.1:52402/_/collection1 node=127.0.0.1:52402__ C715_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:52402/_, core=collection1, node_name=127.0.0.1:52402__, leader=true}
   [junit4]   2> 2828364 T4119 C715 P52402 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2828366 T4119 C715 P52402 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2828367 T4119 C715 P52402 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 2828367 T4119 C715 P52402 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 2828368 T4119 C715 P52402 oasup.LogUpdateProcessor.finish [collection1] webapp=/_ path=/update params={softCommit=false&commit=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 4
   [junit4]   2> 2828389 T4158 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=52417,localport=52407], receiveBufferSize:6565888
   [junit4]   2> 2828390 T4158 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=52408,localport=52418], receiveBufferSize=6565888
   [junit4]   2> 2828393 T4158 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=52419,localport=52407], receiveBufferSize:6565888
   [junit4]   2> 2828393 T4158 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=52408,localport=52420], receiveBufferSize=6565888
   [junit4]   2> ASYNC  NEW_CORE C716 name=collection1 org.apache.solr.core.SolrCore@471bc16a url=http://127.0.0.1:52407/_/collection1 node=127.0.0.1:52407__ C716_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:52407/_, core=collection1, node_name=127.0.0.1:52407__, leader=true}
   [junit4]   2> 2828394 T4143 C716 P52407 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2828394 T4143 C716 P52407 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2828395 T4143 C716 P52407 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 2828395 T4143 C716 P52407 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 2828396 T4143 C716 P52407 oasup.LogUpdateProcessor.finish [collection1] webapp=/_ path=/update params={softCommit=false&distrib.from=http://127.0.0.1:52407/_/collection1/&commit=true&commit_end_point=true&wt=javabin&expungeDeletes=false&update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&version=2} {commit=} 0 2
   [junit4]   2> 2828497 T4177 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=52421,localport=52411], receiveBufferSize:6565888
   [junit4]   2> 2828498 T4177 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=52412,localport=52422], receiveBufferSize=6565888
   [junit4]   2> ASYNC  NEW_CORE C717 name=collection1 org.apache.solr.core.SolrCore@5413c6d8 url=http://127.0.0.1:52411/_/collectio

[...truncated too long message...]

.java:505)
   [junit4]   2> 	   5) Thread[id=1020, name=coreZkRegister-614-thread-1, state=WAITING, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	   6) Thread[id=1017, name=searcherExecutor-619-thread-1, state=WAITING, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	   7) Thread[id=1032, name=parallelCoreAdminExecutor-617-thread-1, state=WAITING, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	   8) Thread[id=1146, name=searcherExecutor-643-thread-1, state=WAITING, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	   9) Thread[id=1141, name=parallelCoreAdminExecutor-617-thread-2, state=WAITING, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  10) Thread[id=1010, name=TEST-CollectionsAPIAsyncDistributedZkTest.testDistribSearch-seed#[84A7796AF29B22FB]-SendThread(127.0.0.1:41431), state=RUNNABLE, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:205)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:243)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:102)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 	  11) Thread[id=940, name=TEST-CollectionsAPIAsyncDistributedZkTest.testDistribSearch-seed#[84A7796AF29B22FB]-SendThread(127.0.0.1:41431), state=RUNNABLE, group=TGRP-CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> 	        at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:167)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:108)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=it_IT, timezone=Asia/Dubai
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=13,free=267752352,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [CustomCollectionTest, ChangedSchemaMergeTest, TestFuzzyAnalyzedSuggestions, SolrCmdDistributorTest, SimplePostToolTest, TestSolrXmlPersistor, SuggesterFSTTest, BadComponentTest, TestReplicationHandlerBackup, TestFieldTypeCollectionResource, HdfsBasicDistributedZk2Test, TestSystemIdResolver, SliceStateTest, TestSolrXml, FileUtilsTest, ResponseLogComponentTest, TestPseudoReturnFields, OpenExchangeRatesOrgProviderTest, TestSuggestSpellingConverter, SimpleFacetsTest, ShardSplitTest, RAMDirectoryFactoryTest, TestLMDirichletSimilarityFactory, BadIndexSchemaTest, DocValuesMultiTest, TestBadConfig, PrimitiveFieldTypeTest, SolrCoreCheckLockOnStartupTest, CollectionsAPIAsyncDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIAsyncDistributedZkTest -Dtests.seed=84A7796AF29B22FB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=Asia/Dubai -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | CollectionsAPIAsyncDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([84A7796AF29B22FB]:0)
   [junit4] Completed on J1 in 7223.81s, 1 test, 2 errors <<< FAILURES!

[...truncated 230 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:467: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1299: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:923: There were test failures: 404 suites, 1654 tests, 1 suite-level error, 2 errors, 333 ignored (13 assumptions)

Total time: 169 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-4.x-Java7 #2008
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 137431407 bytes
Compression is 0.0%
Took 24 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message