lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1430 - Failure
Date Sun, 21 Jul 2013 08:25:13 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1430/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at java.net.PlainSocketImpl.socketConnect(Native Method)         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)         at java.net.Socket.connect(Socket.java:579)         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([65F33D8DB60B35DB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at java.net.PlainSocketImpl.socketConnect(Native Method)         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)         at java.net.Socket.connect(Socket.java:579)         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([65F33D8DB60B35DB]:0)




Build Log:
[...truncated 9289 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 33342 T33 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 33351 T33 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374394190468
   [junit4]   2> 33362 T33 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 33372 T34 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 33574 T33 oasc.ZkTestServer.run start zk server on port:51988
   [junit4]   2> 33878 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 33928 T35 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 35146 T39 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fce884e name:ZooKeeperConnection Watcher:127.0.0.1:51988 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 35147 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 35153 T33 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 35304 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 35307 T42 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69677bba name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 35308 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 35323 T33 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 35331 T33 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 35337 T33 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 35342 T33 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 35348 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 35392 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 35403 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 35404 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 35511 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 35512 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 35518 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 35519 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 35525 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 35526 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 35531 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 35532 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 35538 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 35539 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 35545 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 35546 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 35551 T33 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 35552 T33 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 36548 T33 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 36828 T33 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15284
   [junit4]   2> 36874 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 36875 T33 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 36875 T33 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675
   [junit4]   2> 36876 T33 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/'
   [junit4]   2> 36904 T33 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/solr.xml
   [junit4]   2> 37038 T33 oasc.CoreContainer.<init> New CoreContainer 448864082
   [junit4]   2> 37039 T33 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/]
   [junit4]   2> 37093 T33 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 37093 T33 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 37094 T33 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 37094 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 37094 T33 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 37095 T33 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 37095 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 37096 T33 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 37096 T33 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 37096 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 37110 T33 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 37111 T33 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51988/solr
   [junit4]   2> 37137 T33 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 37138 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37143 T54 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e39c2c0 name:ZooKeeperConnection Watcher:127.0.0.1:51988 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37144 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37155 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 37177 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37180 T56 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21ded493 name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37180 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37198 T33 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 37220 T33 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 37250 T33 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 37254 T33 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15284_
   [junit4]   2> 37265 T33 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15284_
   [junit4]   2> 37271 T33 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 37309 T33 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 37314 T33 oasc.Overseer.start Overseer (id=90072297694887939-127.0.0.1:15284_-n_0000000000) starting
   [junit4]   2> 37352 T33 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 37373 T58 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 37373 T33 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 37377 T33 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 37388 T33 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 37417 T57 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 37438 T59 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 37438 T59 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 37440 T59 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 38930 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 38938 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15284_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15284"}
   [junit4]   2> 38938 T57 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
   [junit4]   2> 38973 T57 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 39002 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 39443 T59 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 39443 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/collection1
   [junit4]   2> 39444 T59 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 39445 T59 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 39446 T59 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 39448 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/collection1/'
   [junit4]   2> 39450 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/collection1/lib/README' to classloader
   [junit4]   2> 39451 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/collection1/lib/classes/' to classloader
   [junit4]   2> 39565 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 39639 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 39742 T59 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 39786 T59 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 40497 T59 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 40498 T59 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 40500 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 40522 T59 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 40527 T59 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 40568 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 40576 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 40582 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 40585 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 40585 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 40585 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 40588 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 40588 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 40589 T59 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 40589 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374394192675/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/control/data/
   [junit4]   2> 40590 T59 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d72cf
   [junit4]   2> 40591 T59 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/control/data
   [junit4]   2> 40592 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/control/data/index/
   [junit4]   2> 40592 T59 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 40593 T59 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/control/data/index
   [junit4]   2> 40599 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8c4eceb lockFactory=org.apache.lucene.store.NativeFSLockFactory@40b7fe45),segFN=segments_1,generation=1}
   [junit4]   2> 40599 T59 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 40681 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 40682 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 40684 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 40684 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 40685 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 40686 T59 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 40686 T59 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 40687 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 40688 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 40704 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 40712 T59 oass.SolrIndexSearcher.<init> Opening Searcher@2fc4a239 main
   [junit4]   2> 40714 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 40715 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 40720 T60 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fc4a239 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 40723 T59 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 40723 T59 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15284 collection:control_collection shard:shard1
   [junit4]   2> 40745 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 40767 T59 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 40798 T59 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 40803 T59 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 40804 T59 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 40804 T59 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15284/collection1/
   [junit4]   2> 40805 T59 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 40805 T59 oasc.SyncStrategy.syncToMe http://127.0.0.1:15284/collection1/ has no replicas
   [junit4]   2> 40806 T59 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15284/collection1/ shard1
   [junit4]   2> 40806 T59 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 42013 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 42033 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 42053 T59 oasc.ZkController.register We are http://127.0.0.1:15284/collection1/ and leader is http://127.0.0.1:15284/collection1/
   [junit4]   2> 42053 T59 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15284
   [junit4]   2> 42053 T59 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 42054 T59 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 42054 T59 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 42057 T59 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 42059 T33 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 42060 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 42157 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 42191 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 42199 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 42203 T63 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25b91df9 name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42203 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 42206 T33 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 42213 T33 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 42501 T33 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 42504 T33 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43415
   [junit4]   2> 42505 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 42506 T33 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 42507 T33 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329
   [junit4]   2> 42507 T33 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/'
   [junit4]   2> 42547 T33 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/solr.xml
   [junit4]   2> 42644 T33 oasc.CoreContainer.<init> New CoreContainer 278813641
   [junit4]   2> 42644 T33 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/]
   [junit4]   2> 42647 T33 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 42647 T33 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 42648 T33 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 42648 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 42649 T33 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 42650 T33 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 42650 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 42651 T33 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 42651 T33 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 42652 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 42670 T33 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 42671 T33 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51988/solr
   [junit4]   2> 42671 T33 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 42673 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 42676 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5247710 name:ZooKeeperConnection Watcher:127.0.0.1:51988 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42677 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 42695 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 42708 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 42711 T76 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36f94f72 name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42711 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 42723 T33 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 43544 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 43547 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15284_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15284"}
   [junit4]   2> 43564 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 43565 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 43565 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 43730 T33 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43415_
   [junit4]   2> 43733 T33 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43415_
   [junit4]   2> 43736 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 43736 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 43736 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 43740 T63 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 43740 T56 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 43741 T76 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 43754 T77 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 43754 T77 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 43756 T77 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 45073 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 45076 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43415_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43415"}
   [junit4]   2> 45076 T57 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
   [junit4]   2> 45076 T57 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 45093 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 45093 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 45093 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 45759 T77 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 45759 T77 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/collection1
   [junit4]   2> 45759 T77 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 45761 T77 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 45761 T77 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 45762 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/collection1/'
   [junit4]   2> 45764 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/collection1/lib/classes/' to classloader
   [junit4]   2> 45764 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/collection1/lib/README' to classloader
   [junit4]   2> 45808 T77 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 45862 T77 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 45965 T77 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 45983 T77 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 46517 T77 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 46518 T77 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 46520 T77 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 46540 T77 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 46545 T77 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 46580 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 46588 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 46594 T77 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 46596 T77 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 46597 T77 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 46597 T77 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 46599 T77 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 46600 T77 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 46600 T77 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 46601 T77 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374394199329/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty1/
   [junit4]   2> 46601 T77 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d72cf
   [junit4]   2> 46602 T77 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty1
   [junit4]   2> 46603 T77 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty1/index/
   [junit4]   2> 46603 T77 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 46604 T77 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty1/index
   [junit4]   2> 46609 T77 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@69885416 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3885bd21),segFN=segments_1,generation=1}
   [junit4]   2> 46609 T77 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 46617 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 46618 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 46619 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 46620 T77 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 46621 T77 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 46621 T77 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 46622 T77 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 46623 T77 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 46624 T77 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 46641 T77 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46650 T77 oass.SolrIndexSearcher.<init> Opening Searcher@139290c9 main
   [junit4]   2> 46651 T77 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 46652 T77 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 46732 T78 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@139290c9 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 46735 T77 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 46735 T77 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43415 collection:collection1 shard:shard1
   [junit4]   2> 46736 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 46751 T77 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 46796 T77 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 46808 T77 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 46808 T77 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 46809 T77 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43415/collection1/
   [junit4]   2> 46809 T77 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 46810 T77 oasc.SyncStrategy.syncToMe http://127.0.0.1:43415/collection1/ has no replicas
   [junit4]   2> 46810 T77 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43415/collection1/ shard1
   [junit4]   2> 46810 T77 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 48105 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 48126 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 48126 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 48126 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 48158 T77 oasc.ZkController.register We are http://127.0.0.1:43415/collection1/ and leader is http://127.0.0.1:43415/collection1/
   [junit4]   2> 48158 T77 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43415
   [junit4]   2> 48159 T77 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 48159 T77 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 48159 T77 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 48162 T77 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 48164 T33 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 48164 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 48165 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 48369 T33 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 48372 T33 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:14958
   [junit4]   2> 48373 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 48373 T33 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 48373 T33 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290
   [junit4]   2> 48374 T33 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/'
   [junit4]   2> 48401 T33 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/solr.xml
   [junit4]   2> 48480 T33 oasc.CoreContainer.<init> New CoreContainer 631269140
   [junit4]   2> 48480 T33 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/]
   [junit4]   2> 48483 T33 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 48483 T33 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 48484 T33 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 48484 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 48485 T33 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 48486 T33 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 48486 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 48487 T33 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 48487 T33 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 48488 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 48509 T33 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 48510 T33 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51988/solr
   [junit4]   2> 48511 T33 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 48512 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 48516 T90 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64037843 name:ZooKeeperConnection Watcher:127.0.0.1:51988 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 48517 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 48531 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 48546 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 48549 T92 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eaaa6a4 name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 48550 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 48558 T33 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 49564 T33 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14958_
   [junit4]   2> 49570 T33 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:14958_
   [junit4]   2> 49573 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 49574 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 49574 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 49575 T92 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 49577 T63 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 49577 T76 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 49578 T56 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 49592 T93 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 49592 T93 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 49594 T93 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 49641 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 49644 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43415_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43415"}
   [junit4]   2> 49652 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:14958_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:14958"}
   [junit4]   2> 49653 T57 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 49653 T57 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 49664 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 49664 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 49664 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 49664 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 50595 T93 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 50595 T93 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/collection1
   [junit4]   2> 50595 T93 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 50597 T93 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 50597 T93 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 50599 T93 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/collection1/'
   [junit4]   2> 50600 T93 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/collection1/lib/README' to classloader
   [junit4]   2> 50600 T93 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/collection1/lib/classes/' to classloader
   [junit4]   2> 50644 T93 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 50714 T93 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 50818 T93 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 50845 T93 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 51520 T93 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 51520 T93 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 51522 T93 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 51544 T93 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 51548 T93 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 51578 T93 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 51584 T93 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 51589 T93 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 51591 T93 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 51591 T93 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 51592 T93 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 51593 T93 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 51594 T93 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 51594 T93 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 51594 T93 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374394205290/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty2/
   [junit4]   2> 51595 T93 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d72cf
   [junit4]   2> 51596 T93 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty2
   [junit4]   2> 51596 T93 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty2/index/
   [junit4]   2> 51596 T93 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 51597 T93 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty2/index
   [junit4]   2> 51600 T93 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@661261ba lockFactory=org.apache.lucene.store.NativeFSLockFactory@48541013),segFN=segments_1,generation=1}
   [junit4]   2> 51600 T93 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 51606 T93 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 51606 T93 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 51607 T93 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 51607 T93 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 51608 T93 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 51608 T93 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 51609 T93 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 51609 T93 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 51610 T93 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 51621 T93 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 51627 T93 oass.SolrIndexSearcher.<init> Opening Searcher@47e4b801 main
   [junit4]   2> 51628 T93 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 51629 T93 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 51633 T94 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47e4b801 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 51636 T93 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 51637 T93 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:14958 collection:collection1 shard:shard2
   [junit4]   2> 51637 T93 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 51651 T93 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 51687 T93 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 51699 T93 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 51699 T93 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 51699 T93 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:14958/collection1/
   [junit4]   2> 51700 T93 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 51700 T93 oasc.SyncStrategy.syncToMe http://127.0.0.1:14958/collection1/ has no replicas
   [junit4]   2> 51700 T93 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:14958/collection1/ shard2
   [junit4]   2> 51700 T93 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 52676 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52694 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52694 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52694 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52694 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 52737 T93 oasc.ZkController.register We are http://127.0.0.1:14958/collection1/ and leader is http://127.0.0.1:14958/collection1/
   [junit4]   2> 52737 T93 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:14958
   [junit4]   2> 52737 T93 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 52737 T93 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 52738 T93 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 52740 T93 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52741 T33 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 52742 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 52743 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 52954 T33 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 52957 T33 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60546
   [junit4]   2> 52957 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 52958 T33 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 52958 T33 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868
   [junit4]   2> 52959 T33 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/'
   [junit4]   2> 52989 T33 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/solr.xml
   [junit4]   2> 53065 T33 oasc.CoreContainer.<init> New CoreContainer 289966798
   [junit4]   2> 53065 T33 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/]
   [junit4]   2> 53067 T33 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 53067 T33 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 53068 T33 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 53068 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 53069 T33 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 53069 T33 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 53069 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 53070 T33 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 53070 T33 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 53070 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 53084 T33 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 53085 T33 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51988/solr
   [junit4]   2> 53085 T33 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 53086 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 53089 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a1145c5 name:ZooKeeperConnection Watcher:127.0.0.1:51988 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 53090 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 53105 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 53116 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 53118 T108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33f99548 name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 53119 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 53132 T33 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 54137 T33 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60546_
   [junit4]   2> 54140 T33 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60546_
   [junit4]   2> 54144 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 54145 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 54144 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 54144 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 54145 T108 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 54148 T56 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 54148 T63 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 54155 T76 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 54155 T92 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 54163 T109 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 54163 T109 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 54165 T109 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 54209 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 54211 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:14958_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:14958"}
   [junit4]   2> 54217 T57 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60546_",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60546"}
   [junit4]   2> 54218 T57 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 54218 T57 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 54228 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 54229 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 54229 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 54229 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 54229 T108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 55166 T109 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 55166 T109 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/collection1
   [junit4]   2> 55166 T109 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 55167 T109 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 55168 T109 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 55169 T109 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/collection1/'
   [junit4]   2> 55171 T109 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/collection1/lib/README' to classloader
   [junit4]   2> 55171 T109 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/collection1/lib/classes/' to classloader
   [junit4]   2> 55232 T109 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 55280 T109 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 55382 T109 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 55399 T109 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 55898 T109 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 55898 T109 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 55899 T109 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 55911 T109 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 55914 T109 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 55940 T109 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 55945 T109 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 55951 T109 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 55953 T109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 55953 T109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 55954 T109 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 55955 T109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 55956 T109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 55956 T109 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 55957 T109 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374394209868/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty3/
   [junit4]   2> 55957 T109 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d72cf
   [junit4]   2> 55958 T109 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty3
   [junit4]   2> 55959 T109 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty3/index/
   [junit4]   2> 55959 T109 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 55960 T109 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374394190467/jetty3/index
   [junit4]   2> 55964 T109 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61eed1f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@57eb8cfa),segFN=segments_1,generation=1}
   [junit4]   2> 55965 T109 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 55973 T109 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 55973 T109 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 55974 T109 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 55975 T109 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 55976 T109 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 55976 T109 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 55977 T109 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 55978 T109 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 55978 T109 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 55996 T109 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 56003 T109 oass.SolrIndexSearcher.<init> Opening Searcher@581f423a main
   [junit4]   2> 56005 T109 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 56005 T109 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 56011 T110 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@581f423a main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 56015 T109 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 56015 T109 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60546 collection:collection1 shard:shard1
   [junit4]   2> 56015 T109 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 56064 T109 oasc.ZkController.register We are http://127.0.0.1:60546/collection1/ and leader is http://127.0.0.1:43415/collection1/
   [junit4]   2> 56065 T109 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60546
   [junit4]   2> 56065 T109 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 56065 T109 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C35 name=collection1 org.apache.solr.core.SolrCore@264ccd21 url=http://127.0.0.1:60546/collection1 node=127.0.0.1:60546_ C35_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:60546_, base_url=http://127.0.0.1:60546}
   [junit4]   2> 56073 T111 C35 P60546 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 56074 T109 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 56074 T111 C35 P60546 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 56075 T111 C35 P60546 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 56075 T111 C35 P60546 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 56076 T33 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 56076 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 56077 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 56077 T111 C35 P60546 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 56372 T33 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 56375 T33 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:58277
   [junit4]   2> 56375 T33 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 56376 T33 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 56376 T33 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374394213205
   [junit4]   2> 56376 T33 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374394213205/'
   [junit4]   2> 56415 T33 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374394213205/solr.xml
   [junit4]   2> 56505 T33 oasc.CoreContainer.<init> New CoreContainer 524466606
   [junit4]   2> 56506 T33 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374394213205/]
   [junit4]   2> 56507 T33 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 56508 T33 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 56508 T33 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 56508 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 56509 T33 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 56509 T33 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 56510 T33 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 56510 T33 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 56510 T33 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 56511 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 56527 T33 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 56528 T33 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51988/solr
   [junit4]   2> 56528 T33 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 56529 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 56532 T123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@165973f5 name:ZooKeeperConnection Watcher:127.0.0.1:51988 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 56533 T33 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 56550 T33 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 56565 T33 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 56567 T125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@747172df name:ZooKeeperConnection Watcher:127.0.0.1:51988/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 56567 T33 oascc.Connect

[...truncated too long message...]

ncStrategy.java:291)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:722)
   [junit4]   2> 393049 T32 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 396057 T32 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   [junit4]   2> 	        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
   [junit4]   2> 	        at java.net.Socket.connect(Socket.java:579)
   [junit4]   2> 	        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]   2> 	        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]   2> 	        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]   2> 	        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]   2> 	        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]   2> 	        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]   2> 	        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]   2> 	        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]   2> 	        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]   2> 	        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]   2> 	        at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:722)
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_EC, timezone=Egypt
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=293395384,total=400556032
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastWriter, RegexBoostProcessorTest, IndexReaderFactoryTest, TestFunctionQuery, StatsComponentTest, BasicDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=65F33D8DB60B35DB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_EC -Dtests.timezone=Egypt -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]    >    1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   [junit4]    >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
   [junit4]    >         at java.net.Socket.connect(Socket.java:579)
   [junit4]    >         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    >         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    >         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    >         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:722)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([65F33D8DB60B35DB]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=492, name=recoveryCmdExecutor-76-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   [junit4]    >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
   [junit4]    >         at java.net.Socket.connect(Socket.java:579)
   [junit4]    >         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    >         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    >         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    >         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:722)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([65F33D8DB60B35DB]:0)
   [junit4] Completed on J0 in 396.20s, 1 test, 2 errors <<< FAILURES!

[...truncated 890 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:395: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:375: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1250: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:893: There were test failures: 319 suites, 1345 tests, 2 suite-level errors, 921 ignored (5 assumptions)

Total time: 37 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message