lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b07) - Build # 9715 - Failure!
Date Mon, 17 Mar 2014 05:12:52 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9715/
Java: 32bit/jdk1.7.0_60-ea-b07 -server -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:54873 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:54873 within 45000 ms
	at __randomizedtesting.SeedInfo.seed([C998591B965BEF7:EFEF8C045E1F2B85]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:54873 within 45000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 45 more




Build Log:
[...truncated 11780 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> 1009 T10 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1035 T10 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1039 T10 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 1071 T10 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395033038106
   [junit4]   2> 1075 T10 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1079 T11 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1178 T10 oasc.ZkTestServer.run start zk server on port:54873
   [junit4]   2> 1228 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 54857 T14 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 48306ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 54870 T12 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144ce72dd7d0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 54875 T12 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 54876 T12 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 54908 T10 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 54911 T10 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54873 54873
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=C998591B965BEF7 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es -Dtests.timezone=Africa/Brazzaville -Dtests.file.encoding=UTF-8
   [junit4] ERROR   54.3s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:54873 within 45000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C998591B965BEF7:EFEF8C045E1F2B85]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:54873 within 45000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
   [junit4]    > 	... 45 more
   [junit4]   2> 55358 T10 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395033092393
   [junit4]   2> 55359 T10 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 55361 T18 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 55461 T10 oasc.ZkTestServer.run start zk server on port:37063
   [junit4]   2> 55463 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55490 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ce4360 name:ZooKeeperConnection Watcher:127.0.0.1:37063 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55491 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55493 T10 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 55524 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55525 T26 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dc7b4e name:ZooKeeperConnection Watcher:127.0.0.1:37063/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55526 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55536 T10 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 55556 T10 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 55582 T10 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 55588 T10 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 55595 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 55608 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 55619 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 55620 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 55658 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 55659 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 55660 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 55660 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 55661 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 55662 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 55663 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 55664 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 55665 T10 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 55962 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 56053 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36228
   [junit4]   2> 56147 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 56182 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 56182 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702
   [junit4]   2> 56183 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702/'
   [junit4]   2> 56329 T10 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702/solr.xml
   [junit4]   2> 56524 T10 oasc.CoreContainer.<init> New CoreContainer 239843
   [junit4]   2> 56525 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702/]
   [junit4]   2> 56552 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 56552 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 56561 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 56562 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 56562 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 56563 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 56564 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 56565 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 56565 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 56785 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 56788 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 56791 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 56792 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 56802 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 56803 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 56808 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d8a434 name:ZooKeeperConnection Watcher:127.0.0.1:37063 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 56809 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 56815 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 56817 T40 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e54be2 name:ZooKeeperConnection Watcher:127.0.0.1:37063/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 56818 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 56824 T10 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 56836 T10 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 56846 T10 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 56850 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36228_
   [junit4]   2> 56853 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36228_
   [junit4]   2> 56859 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 56863 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 56874 T10 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36228_
   [junit4]   2> 56874 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 56880 T10 oasc.Overseer.start Overseer (id=91424888743395331-127.0.0.1:36228_-n_0000000000) starting
   [junit4]   2> 56893 T10 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 57031 T42 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 57032 T10 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 57039 T10 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 57043 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 57046 T42 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 57060 T41 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 57176 T43 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 57177 T43 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 57181 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 57182 T43 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 57185 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 57187 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36228",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36228_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 57187 T41 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 57198 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 57204 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 57209 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 58183 T43 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 58183 T43 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702/collection1
   [junit4]   2> 58184 T43 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 58185 T43 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 58186 T43 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 58188 T43 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 58188 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702/collection1/'
   [junit4]   2> 58279 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 58432 T43 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 58440 T43 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 58531 T43 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 58693 T43 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 58705 T43 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 58717 T43 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 58724 T43 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 59119 T43 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 59121 T43 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 59126 T43 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 59126 T43 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 59166 T43 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 59174 T43 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395033092702/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data/
   [junit4]   2> 59176 T43 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 59198 T43 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data
   [junit4]   2> 59198 T43 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data/index/
   [junit4]   2> 59199 T43 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 59208 T43 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data/index
   [junit4]   2> 59250 T43 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b86cb5 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 59251 T43 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 59420 T43 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 59435 T43 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 59435 T43 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 59437 T43 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 59441 T43 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 59447 T43 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 59449 T43 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 59453 T43 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 59483 T43 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 59509 T43 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 59509 T43 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 59511 T43 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b86cb5 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 59511 T43 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 59524 T43 oass.SolrIndexSearcher.<init> Opening Searcher@11d15b6[collection1] main
   [junit4]   2> 59557 T43 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 59557 T44 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11d15b6[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 59560 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 59560 T47 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36228 collection:control_collection shard:shard1
   [junit4]   2> 59561 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 59569 T47 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 59576 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 59584 T49 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29a3c2 name:ZooKeeperConnection Watcher:127.0.0.1:37063/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 59584 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 59586 T47 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 59587 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 59590 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59591 T47 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 59591 T47 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 59592 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 59593 T10 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@1e715fa url=http://127.0.0.1:36228/collection1 node=127.0.0.1:36228_ C0_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:36228, core=collection1, node_name=127.0.0.1:36228_}
   [junit4]   2> 59595 T47 C0 P36228 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36228/collection1/
   [junit4]   2> 59596 T47 C0 P36228 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 59597 T47 C0 P36228 oasc.SyncStrategy.syncToMe http://127.0.0.1:36228/collection1/ has no replicas
   [junit4]   2> 59598 T47 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36228/collection1/ shard1
   [junit4]   2> 59598 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59599 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 59600 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 59602 T47 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 59620 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59623 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 59628 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59641 T10 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 59644 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 59646 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53212
   [junit4]   2> 59647 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 59647 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 59648 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627
   [junit4]   2> 59648 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627/'
   [junit4]   2> 59661 T10 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627/solr.xml
   [junit4]   2> 59716 T10 oasc.CoreContainer.<init> New CoreContainer 6719157
   [junit4]   2> 59716 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627/]
   [junit4]   2> 59718 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 59718 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 59719 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 59719 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 59719 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 59720 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 59720 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 59721 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 59721 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 59722 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 59722 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 59723 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 59723 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 59724 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 59725 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 59727 T60 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@115e196 name:ZooKeeperConnection Watcher:127.0.0.1:37063 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 59730 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 59731 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 59731 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 59735 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 59740 T62 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@145de8a name:ZooKeeperConnection Watcher:127.0.0.1:37063/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 59740 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 59746 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 59772 T47 oasc.ZkController.register We are http://127.0.0.1:36228/collection1/ and leader is http://127.0.0.1:36228/collection1/
   [junit4]   2> 59774 T47 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36228
   [junit4]   2> 59774 T47 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 59775 T47 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 59775 T47 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 59777 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59778 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59778 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59778 T47 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 59780 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 59782 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36228",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36228_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 59786 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59888 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 59889 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 59889 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 60751 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53212_
   [junit4]   2> 60753 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53212_
   [junit4]   2> 60756 T49 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 60756 T40 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 60757 T62 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 60767 T63 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 60767 T63 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 60769 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 60769 T63 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 60770 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 60770 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 60773 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 60774 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53212",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53212_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 60775 T41 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 60775 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 60779 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 60883 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 60883 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 60884 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 61770 T63 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 61771 T63 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627/collection1
   [junit4]   2> 61771 T63 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 61772 T63 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 61772 T63 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 61774 T63 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 61774 T63 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627/collection1/'
   [junit4]   2> 61812 T63 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 61853 T63 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 61855 T63 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 61921 T63 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 62229 T63 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 62232 T63 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 62236 T63 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 62237 T63 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 62239 T63 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 62239 T63 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395033096627/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1/
   [junit4]   2> 62240 T63 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 62241 T63 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1
   [junit4]   2> 62241 T63 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1/index/
   [junit4]   2> 62242 T63 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 62243 T63 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1/index
   [junit4]   2> 62245 T63 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9c1e43 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 62245 T63 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 62247 T63 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 62248 T63 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 62248 T63 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 62249 T63 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 62249 T63 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 62250 T63 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 62250 T63 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 62250 T63 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 62253 T63 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 62256 T63 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 62256 T63 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 62257 T63 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9c1e43 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 62257 T63 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 62258 T63 oass.SolrIndexSearcher.<init> Opening Searcher@1db7247[collection1] main
   [junit4]   2> 62260 T64 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1db7247[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 62260 T63 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 62261 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 62261 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 62263 T67 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53212 collection:collection1 shard:shard2
   [junit4]   2> 62266 T67 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 62278 T67 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 62286 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62287 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62287 T67 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 62287 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62288 T67 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 62289 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@11c4e54 url=http://127.0.0.1:53212/collection1 node=127.0.0.1:53212_ C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53212, core=collection1, node_name=127.0.0.1:53212_}
   [junit4]   2> 62290 T67 C1 P53212 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53212/collection1/
   [junit4]   2> 62291 T67 C1 P53212 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 62291 T67 C1 P53212 oasc.SyncStrategy.syncToMe http://127.0.0.1:53212/collection1/ has no replicas
   [junit4]   2> 62291 T67 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53212/collection1/ shard2
   [junit4]   2> 62292 T67 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 62299 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62303 T10 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 62304 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 62305 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62306 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58177
   [junit4]   2> 62307 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 62307 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 62308 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296
   [junit4]   2> 62308 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296/'
   [junit4]   2> 62316 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62321 T10 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296/solr.xml
   [junit4]   2> 62355 T10 oasc.CoreContainer.<init> New CoreContainer 26245332
   [junit4]   2> 62356 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296/]
   [junit4]   2> 62358 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 62358 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 62359 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 62359 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 62360 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 62361 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 62361 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 62362 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 62362 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 62363 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 62364 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 62364 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 62365 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 62365 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 62366 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 62368 T78 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ecda63 name:ZooKeeperConnection Watcher:127.0.0.1:37063 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62369 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 62373 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 62376 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1daeb4d name:ZooKeeperConnection Watcher:127.0.0.1:37063/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62376 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 62384 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 62422 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62422 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62422 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62422 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62460 T67 oasc.ZkController.register We are http://127.0.0.1:53212/collection1/ and leader is http://127.0.0.1:53212/collection1/
   [junit4]   2> 62461 T67 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53212
   [junit4]   2> 62461 T67 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 62461 T67 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 62462 T67 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 62463 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62464 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62464 T67 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 62464 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62466 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 62468 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53212",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53212_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 62472 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 62575 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62575 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62575 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 62575 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 63389 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58177_
   [junit4]   2> 63391 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58177_
   [junit4]   2> 63394 T62 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 63395 T49 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 63395 T40 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 63395 T80 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 63406 T81 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 63407 T81 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 63409 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 63409 T81 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 63409 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 63410 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 63411 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 63413 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58177",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58177_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 63413 T41 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 63413 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 63417 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 63520 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 63520 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 63520 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 63520 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 64410 T81 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 64410 T81 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296/collection1
   [junit4]   2> 64410 T81 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 64411 T81 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 64412 T81 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 64415 T81 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 64415 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296/collection1/'
   [junit4]   2> 64457 T81 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 64492 T81 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 64494 T81 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 64589 T81 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 64885 T81 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 64888 T81 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 64892 T81 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 64892 T81 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 64894 T81 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 64895 T81 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395033099296/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2/
   [junit4]   2> 64896 T81 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 64897 T81 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2
   [junit4]   2> 64897 T81 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2/index/
   [junit4]   2> 64897 T81 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 64898 T81 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2/index
   [junit4]   2> 64900 T81 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5d1a72 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 64900 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 64903 T81 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 64903 T81 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 64903 T81 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 64904 T81 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 64904 T81 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 64904 T81 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 64905 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 64905 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 64907 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 64909 T81 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 64910 T81 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 64911 T81 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5d1a72 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 64911 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 64911 T81 oass.SolrIndexSearcher.<init> Opening Searcher@1b208de[collection1] main
   [junit4]   2> 64912 T82 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b208de[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 64912 T81 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 64913 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 64914 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 64913 T85 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58177 collection:collection1 shard:shard1
   [junit4]   2> 64916 T85 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 64926 T85 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 64929 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 64929 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 64929 T85 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 64930 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 64930 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@504e1f url=http://127.0.0.1:58177/collection1 node=127.0.0.1:58177_ C2_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:58177, core=collection1, node_name=127.0.0.1:58177_}
   [junit4]   2> 64931 T85 C2 P58177 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58177/collection1/
   [junit4]   2> 64931 T85 C2 P58177 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 64931 T85 C2 P58177 oasc.SyncStrategy.syncToMe http://127.0.0.1:58177/collection1/ has no replicas
   [junit4]   2> 64932 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58177/collection1/ shard1
   [junit4]   2> 64932 T85 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 64932 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 64937 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 64939 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 64942 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 64952 T10 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 64953 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 64955 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53960
   [junit4]   2> 64956 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 64957 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 64957 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949
   [junit4]   2> 64958 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949/'
   [junit4]   2> 64975 T10 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949/solr.xml
   [junit4]   2> 65005 T10 oasc.CoreContainer.<init> New CoreContainer 17121463
   [junit4]   2> 65005 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949/]
   [junit4]   2> 65006 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 65007 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 65007 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 65008 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 65008 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 65008 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 65009 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 65009 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 65010 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 65014 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 65014 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 65014 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 65015 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37063/solr
   [junit4]   2> 65015 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 65017 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 65019 T96 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dbc992 name:ZooKeeperConnection Watcher:127.0.0.1:37063 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65020 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 65024 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 65025 T98 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@178c35 name:ZooKeeperConnection Watcher:127.0.0.1:37063/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65025 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 65031 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 65077 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65077 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65077 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65077 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65078 T98 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65090 T85 oasc.ZkController.register We are http://127.0.0.1:58177/collection1/ and leader is http://127.0.0.1:58177/collection1/
   [junit4]   2> 65090 T85 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58177
   [junit4]   2> 65090 T85 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 65091 T85 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 65091 T85 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 65093 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 65093 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 65094 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 65094 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 65096 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 65098 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58177",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58177_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 65102 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 65206 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65206 T98 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65206 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65206 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 65206 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 66035 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53960_
   [junit4]   2> 66037 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53960_
   [junit4]   2> 66040 T98 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 66041 T49 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 66041 T40 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 66041 T62 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 66041 T80 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 66049 T99 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 66049 T99 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 66051 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 66051 T99 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 66051 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 66052 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 66054 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 66055 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53960",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53960_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 66055 T41 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 66055 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 66059 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 66162 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 66162 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 66162 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 66162 T98 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 66162 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 67051 T99 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 67052 T99 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949/collection1
   [junit4]   2> 67052 T99 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 67053 T99 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 67053 T99 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 67055 T99 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 67055 T99 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949/collection1/'
   [junit4]   2> 67087 T99 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 67117 T99 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 67119 T99 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 67165 T99 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 67394 T99 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 67396 T99 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 67398 T99 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 67398 T99 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 67399 T99 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 67400 T99 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395033101949/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/
   [junit4]   2> 67400 T99 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 67401 T99 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3
   [junit4]   2> 67401 T99 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index/
   [junit4]   2> 67401 T99 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 67402 T99 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index
   [junit4]   2> 67403 T99 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1e7b06c lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 67403 T99 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 67404 T99 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 67405 T99 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 67405 T99 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 67405 T99 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 67406 T99 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 67406 T99 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 67406 T99 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 67407 T99 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 67409 T99 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 67411 T99 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 67411 T99 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 67412 T99 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1e7b06c lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 67412 T99 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 67413 T99 oass.SolrIndexSearcher.<init> Opening Searcher@f7f655[collection1] main
   [junit4]   2> 67415 T100 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f7f655[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 67415 T99 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 67416 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 67416 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 67416 T103 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53960 collection:collection1 shard:shard2
   [junit4]   2> 67421 T103 oasc.ZkController.register We are http://127.0.0.1:53960/collection1/ and leader is http://127.0.0.1:53212/collection1/
   [junit4]   2> 67422 T103 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53960
   [junit4]   2> 67422 T103 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 67422 T103 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@172b4a6 url=http://127.0.0.1:53960/collection1 node=127.0.0.1:53960_ C3_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53960, core=collection1, node_name=127.0.0.1:53960_}
   [junit4]   2> 67425 T104 C3 P53960 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 67426 T104 C3 P53960 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 67426 T103 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 67427 T104 C3 P53960 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 67427 T104 C3 P53960 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 67429 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 67429 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 67430 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 67432 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 67433 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53960",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53960_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_n

[...truncated too long message...]

o 127.0.0.1:37063 37063
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 79740 T10 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 80730 T40 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80731 T40 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80839 T62 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80939 T80 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 81855 T98 C10 P53960 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 81859 T98 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 81860 T98 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@172b4a6
   [junit4]   2> 81860 T98 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 81861 T98 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 81861 T98 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 81861 T98 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 81862 T98 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 81863 T98 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 81863 T98 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index;done=false>>]
   [junit4]   2> 81863 T98 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3/index
   [junit4]   2> 81864 T98 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3;done=false>>]
   [junit4]   2> 81864 T98 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty3
   [junit4]   2> 81864 T98 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 81865 T98 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 81865 T98 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 81866 T98 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 81944 T117 C6 P37465 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 81945 T117 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 81946 T117 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@23712f
   [junit4]   2> 81946 T117 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 81947 T117 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 81947 T117 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 81947 T117 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 81948 T117 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 81949 T117 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 81949 T117 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty4/index;done=false>>]
   [junit4]   2> 81949 T117 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty4/index
   [junit4]   2> 81950 T117 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty4;done=false>>]
   [junit4]   2> 81950 T117 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395033092393/jetty4
   [junit4]   2> 81951 T117 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 81951 T117 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es, timezone=Africa/Brazzaville
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=38506512,total=187957248
   [junit4]   2> NOTE: All tests run in this JVM: [CloudSolrServerTest]
   [junit4] Completed on J0 in 82.90s, 2 tests, 1 error <<< FAILURES!

[...truncated 124 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 49 suites, 283 tests, 1 error

Total time: 49 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b07 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message