lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_05) - Build # 10327 - Still Failing!
Date Sun, 01 Jun 2014 21:14:07 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10327/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([67CA34973E01DF85]:0)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([67CA34973E01DF85]:0)




Build Log:
[...truncated 12050 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/init-core-data-001
   [junit4]   2> 1075499 T3933 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1075499 T3933 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /iw_w/
   [junit4]   2> 1075503 T3933 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1075503 T3933 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1075504 T3934 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1075604 T3933 oasc.ZkTestServer.run start zk server on port:50587
   [junit4]   2> 1075605 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1075656 T3940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@668cfa3c name:ZooKeeperConnection Watcher:127.0.0.1:50587 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1075657 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1075657 T3933 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1075668 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1075670 T3942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3717c43f name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1075670 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1075670 T3933 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1075673 T3933 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1075674 T3933 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1075675 T3933 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1075676 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1075676 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1075678 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1075679 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1075680 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1075681 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1075682 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1075682 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1075684 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1075684 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1075685 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1075686 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1075687 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1075687 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1075689 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1075689 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1075690 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1075691 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1075696 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1075697 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1075699 T3933 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1075700 T3933 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1075702 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1075703 T3944 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c0c9411 name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1075704 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1075927 T3933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1075930 T3933 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1075932 T3933 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:५४७५९
   [junit4]   2> 1075933 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1075933 T3933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1075933 T3933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002
   [junit4]   2> 1075934 T3933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/'
   [junit4]   2> 1075943 T3933 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/solr.xml
   [junit4]   2> 1075950 T3933 oasc.CoreContainer.<init> New CoreContainer 287910856
   [junit4]   2> 1075951 T3933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/]
   [junit4]   2> 1075951 T3933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1075951 T3933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1075952 T3933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1075952 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1075952 T3933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1075952 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1075953 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1075953 T3933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1075953 T3933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1075954 T3933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1075955 T3933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1075955 T3933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1075955 T3933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50587/solr
   [junit4]   2> 1075956 T3933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1075956 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1075958 T3955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d6caf02 name:ZooKeeperConnection Watcher:127.0.0.1:50587 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1075959 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1075961 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1075962 T3957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78eca68d name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1075963 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1075964 T3933 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1075965 T3933 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1075966 T3933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1075968 T3933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1075969 T3933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1075971 T3933 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1075971 T3933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54759_iw_w
   [junit4]   2> 1075972 T3933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54759_iw_w
   [junit4]   2> 1075973 T3933 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1075974 T3933 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1075981 T3933 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:54759_iw_w
   [junit4]   2> 1075981 T3933 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1075983 T3933 oasc.Overseer.start Overseer (id=91858534034964484-127.0.0.1:54759_iw_w-n_0000000000) starting
   [junit4]   2> 1075985 T3933 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1075990 T3959 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1075990 T3933 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1075992 T3933 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1075993 T3933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1075994 T3958 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1075995 T3960 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1075995 T3960 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1075996 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1075996 T3960 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1075996 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1075997 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:54759/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54759_iw_w",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1075997 T3958 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1075997 T3958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1075998 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1075998 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1076996 T3960 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1076996 T3960 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1076997 T3960 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1076997 T3960 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1076998 T3960 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1076998 T3960 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/collection1/'
   [junit4]   2> 1076999 T3960 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1076999 T3960 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1077000 T3960 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1077022 T3960 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1077033 T3960 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1077034 T3960 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1077037 T3960 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1077135 T3960 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1077136 T3960 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1077137 T3960 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1077139 T3960 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1077149 T3960 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1077150 T3960 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1077150 T3960 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data/
   [junit4]   2> 1077150 T3960 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a34e251
   [junit4]   2> 1077151 T3960 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data
   [junit4]   2> 1077151 T3960 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data/index/
   [junit4]   2> 1077151 T3960 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1077152 T3960 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data/index
   [junit4]   2> 1077152 T3960 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=21, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1742535542940279]
   [junit4]   2> 1077157 T3960 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1077157 T3960 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1077159 T3960 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1077159 T3960 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1077159 T3960 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1077159 T3960 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1077159 T3960 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1077159 T3960 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1077160 T3960 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1077160 T3960 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1077160 T3960 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1077160 T3960 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1077161 T3960 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1077161 T3960 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1077161 T3960 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1077161 T3960 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1077161 T3960 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1077162 T3960 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1077163 T3960 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1077164 T3960 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1077164 T3960 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1077165 T3960 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=22.6181640625, floorSegmentMB=0.2431640625, forceMergeDeletesPctAllowed=9.054725249745584, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1077165 T3960 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1077166 T3960 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1077166 T3960 oass.SolrIndexSearcher.<init> Opening Searcher@6a91d29e[collection1] main
   [junit4]   2> 1077166 T3960 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1077167 T3960 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1077167 T3960 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1077167 T3960 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1077167 T3960 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1077167 T3960 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1077168 T3960 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1077168 T3960 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1077168 T3960 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1077168 T3960 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1077170 T3961 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a91d29e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1077170 T3960 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1077170 T3964 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:54759/iw_w collection:control_collection shard:shard1
   [junit4]   2> 1077171 T3933 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1077171 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1077172 T3964 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1077173 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1077175 T3966 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63624111 name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1077175 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1077176 T3933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1077176 T3964 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1077177 T3933 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1077178 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077178 T3964 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1077178 T3964 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1739 name=collection1 org.apache.solr.core.SolrCore@6b26610c url=https://127.0.0.1:54759/iw_w/collection1 node=127.0.0.1:54759_iw_w C1739_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:54759/iw_w, core=collection1, node_name=127.0.0.1:54759_iw_w}
   [junit4]   2> 1077178 T3964 C1739 P54759 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:54759/iw_w/collection1/
   [junit4]   2> 1077178 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1077178 T3964 C1739 P54759 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1077179 T3964 C1739 P54759 oasc.SyncStrategy.syncToMe https://127.0.0.1:54759/iw_w/collection1/ has no replicas
   [junit4]   2> 1077179 T3964 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:54759/iw_w/collection1/ shard1
   [junit4]   2> 1077179 T3964 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1077180 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077182 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077186 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077288 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1077288 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1077333 T3964 oasc.ZkController.register We are https://127.0.0.1:54759/iw_w/collection1/ and leader is https://127.0.0.1:54759/iw_w/collection1/
   [junit4]   2> 1077334 T3964 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:54759/iw_w
   [junit4]   2> 1077334 T3964 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1077334 T3964 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1077334 T3964 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1077335 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077335 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077335 T3964 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1077335 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077337 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1077337 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:54759/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54759_iw_w",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1077338 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1077440 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1077440 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1077473 T3933 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1077474 T3933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1077476 T3933 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1077477 T3933 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:५९९३९
   [junit4]   2> 1077478 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1077479 T3933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1077479 T3933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003
   [junit4]   2> 1077479 T3933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/'
   [junit4]   2> 1077489 T3933 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/solr.xml
   [junit4]   2> 1077506 T3933 oasc.CoreContainer.<init> New CoreContainer 652774648
   [junit4]   2> 1077506 T3933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/]
   [junit4]   2> 1077506 T3933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1077507 T3933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1077507 T3933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1077507 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1077507 T3933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1077508 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1077508 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1077508 T3933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1077508 T3933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1077510 T3933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1077510 T3933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1077510 T3933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1077511 T3933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50587/solr
   [junit4]   2> 1077511 T3933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1077511 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1077513 T3977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@780d3c91 name:ZooKeeperConnection Watcher:127.0.0.1:50587 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1077514 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1077516 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1077517 T3979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7da422f4 name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1077517 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1077521 T3933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1078523 T3933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59939_iw_w
   [junit4]   2> 1078524 T3933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59939_iw_w
   [junit4]   2> 1078526 T3966 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1078526 T3957 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1078526 T3979 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1078529 T3980 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1078529 T3980 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1078530 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1078530 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1078530 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1078530 T3980 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1078530 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1078531 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59939/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59939_iw_w",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1078531 T3958 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1078531 T3958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1078540 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1078641 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1078641 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1078641 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079530 T3980 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1079531 T3980 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1079531 T3980 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1079531 T3980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1079532 T3980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1079532 T3980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/collection1/'
   [junit4]   2> 1079533 T3980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1079533 T3980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1079533 T3980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1079549 T3980 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1079558 T3980 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1079559 T3980 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1079562 T3980 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1079663 T3980 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1079664 T3980 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1079665 T3980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1079667 T3980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1079677 T3980 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1079677 T3980 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1079677 T3980 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1/
   [junit4]   2> 1079677 T3980 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a34e251
   [junit4]   2> 1079678 T3980 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1
   [junit4]   2> 1079678 T3980 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1/index/
   [junit4]   2> 1079678 T3980 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1079679 T3980 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1/index
   [junit4]   2> 1079679 T3980 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=21, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1742535542940279]
   [junit4]   2> 1079683 T3980 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1079683 T3980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1079685 T3980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1079685 T3980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1079686 T3980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1079686 T3980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1079686 T3980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1079686 T3980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1079686 T3980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1079687 T3980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1079687 T3980 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1079687 T3980 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1079687 T3980 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1079687 T3980 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1079688 T3980 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1079688 T3980 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1079688 T3980 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1079688 T3980 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1079690 T3980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1079691 T3980 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1079691 T3980 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1079691 T3980 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=22.6181640625, floorSegmentMB=0.2431640625, forceMergeDeletesPctAllowed=9.054725249745584, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1079692 T3980 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1079692 T3980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1079692 T3980 oass.SolrIndexSearcher.<init> Opening Searcher@6aae12d[collection1] main
   [junit4]   2> 1079693 T3980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1079693 T3980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1079694 T3980 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1079694 T3980 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1079694 T3980 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1079694 T3980 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1079695 T3980 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1079695 T3980 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1079695 T3980 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1079695 T3980 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1079697 T3981 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6aae12d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1079698 T3980 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1079698 T3984 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59939/iw_w collection:collection1 shard:shard2
   [junit4]   2> 1079699 T3933 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1079699 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1079699 T3984 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1079703 T3984 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1079704 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079704 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079704 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079705 T3984 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1079705 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1079705 T3984 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1740 name=collection1 org.apache.solr.core.SolrCore@59b4ceb2 url=https://127.0.0.1:59939/iw_w/collection1 node=127.0.0.1:59939_iw_w C1740_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:59939/iw_w, core=collection1, node_name=127.0.0.1:59939_iw_w}
   [junit4]   2> 1079705 T3984 C1740 P59939 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59939/iw_w/collection1/
   [junit4]   2> 1079705 T3984 C1740 P59939 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1079706 T3984 C1740 P59939 oasc.SyncStrategy.syncToMe https://127.0.0.1:59939/iw_w/collection1/ has no replicas
   [junit4]   2> 1079706 T3984 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59939/iw_w/collection1/ shard2
   [junit4]   2> 1079706 T3984 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1079707 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079710 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079712 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079817 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079817 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079817 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079861 T3984 oasc.ZkController.register We are https://127.0.0.1:59939/iw_w/collection1/ and leader is https://127.0.0.1:59939/iw_w/collection1/
   [junit4]   2> 1079861 T3984 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59939/iw_w
   [junit4]   2> 1079862 T3984 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1079862 T3984 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1079862 T3984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1079863 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079863 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079863 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079864 T3984 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1079864 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1079864 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59939/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59939_iw_w",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1079866 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1079935 T3933 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1079935 T3933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1079937 T3933 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1079938 T3933 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:६०१२८
   [junit4]   2> 1079940 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1079940 T3933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1079941 T3933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004
   [junit4]   2> 1079941 T3933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/'
   [junit4]   2> 1079951 T3933 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/solr.xml
   [junit4]   2> 1079958 T3933 oasc.CoreContainer.<init> New CoreContainer 497747448
   [junit4]   2> 1079958 T3933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/]
   [junit4]   2> 1079959 T3933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1079959 T3933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1079959 T3933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1079960 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1079960 T3933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1079960 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1079960 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1079960 T3933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1079961 T3933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1079962 T3933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1079962 T3933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1079963 T3933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1079963 T3933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50587/solr
   [junit4]   2> 1079963 T3933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1079964 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1079966 T3995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2de241be name:ZooKeeperConnection Watcher:127.0.0.1:50587 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1079967 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1079967 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079967 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079967 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1079968 T3935 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14658d99abe0008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1079968 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1079971 T3997 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71c897aa name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1079971 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1079975 T3933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1080977 T3933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60128_iw_w
   [junit4]   2> 1080978 T3933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60128_iw_w
   [junit4]   2> 1080981 T3979 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1080981 T3997 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1080981 T3966 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1080981 T3957 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1080987 T3998 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1080988 T3998 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1080988 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1080988 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1080988 T3998 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1080988 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1080989 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1080989 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60128/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60128_iw_w",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1080990 T3958 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1080990 T3958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1080991 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1081093 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1081093 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1081093 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1081093 T3997 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1081989 T3998 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1081989 T3998 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1081989 T3998 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1081989 T3998 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1081990 T3998 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1081990 T3998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/collection1/'
   [junit4]   2> 1081991 T3998 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1081991 T3998 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1081991 T3998 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1082007 T3998 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1082016 T3998 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1082017 T3998 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1082021 T3998 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1082135 T3998 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1082136 T3998 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1082137 T3998 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1082139 T3998 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1082149 T3998 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1082149 T3998 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1082149 T3998 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2/
   [junit4]   2> 1082149 T3998 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a34e251
   [junit4]   2> 1082150 T3998 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2
   [junit4]   2> 1082150 T3998 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2/index/
   [junit4]   2> 1082150 T3998 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1082151 T3998 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2/index
   [junit4]   2> 1082152 T3998 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=21, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1742535542940279]
   [junit4]   2> 1082156 T3998 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1082156 T3998 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1082157 T3998 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1082158 T3998 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1082158 T3998 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1082158 T3998 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1082158 T3998 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1082158 T3998 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1082159 T3998 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1082159 T3998 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1082159 T3998 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1082159 T3998 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1082160 T3998 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1082160 T3998 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1082160 T3998 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1082160 T3998 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1082160 T3998 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1082161 T3998 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1082162 T3998 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1082163 T3998 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1082163 T3998 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1082164 T3998 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=22.6181640625, floorSegmentMB=0.2431640625, forceMergeDeletesPctAllowed=9.054725249745584, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1082164 T3998 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1082165 T3998 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1082165 T3998 oass.SolrIndexSearcher.<init> Opening Searcher@9b64191[collection1] main
   [junit4]   2> 1082165 T3998 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1082165 T3998 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1082166 T3998 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1082166 T3998 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1082166 T3998 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1082166 T3998 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1082167 T3998 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1082167 T3998 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1082167 T3998 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1082167 T3998 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1082168 T3999 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9b64191[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1082169 T3998 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1082170 T4002 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60128/iw_w collection:collection1 shard:shard3
   [junit4]   2> 1082170 T3933 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1082171 T4002 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 1082171 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1082173 T4002 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 1082176 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082176 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082176 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082176 T4002 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1082176 T4002 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1741 name=collection1 org.apache.solr.core.SolrCore@2bf7b039 url=https://127.0.0.1:60128/iw_w/collection1 node=127.0.0.1:60128_iw_w C1741_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:60128/iw_w, core=collection1, node_name=127.0.0.1:60128_iw_w}
   [junit4]   2> 1082177 T4002 C1741 P60128 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60128/iw_w/collection1/
   [junit4]   2> 1082177 T4002 C1741 P60128 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1082177 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1082177 T4002 C1741 P60128 oasc.SyncStrategy.syncToMe https://127.0.0.1:60128/iw_w/collection1/ has no replicas
   [junit4]   2> 1082177 T4002 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60128/iw_w/collection1/ shard3
   [junit4]   2> 1082177 T4002 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 1082179 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082181 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082183 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082285 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082285 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082285 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082285 T3997 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082332 T4002 oasc.ZkController.register We are https://127.0.0.1:60128/iw_w/collection1/ and leader is https://127.0.0.1:60128/iw_w/collection1/
   [junit4]   2> 1082332 T4002 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60128/iw_w
   [junit4]   2> 1082332 T4002 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1082332 T4002 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1082332 T4002 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1082334 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082334 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082334 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082334 T4002 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1082334 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1082335 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60128/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60128_iw_w",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1082336 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1082410 T3933 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1082411 T3933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1082413 T3933 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1082414 T3933 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:५०७७५
   [junit4]   2> 1082416 T3933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1082416 T3933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1082416 T3933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005
   [junit4]   2> 1082416 T3933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/'
   [junit4]   2> 1082426 T3933 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/solr.xml
   [junit4]   2> 1082433 T3933 oasc.CoreContainer.<init> New CoreContainer 1098113219
   [junit4]   2> 1082434 T3933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/]
   [junit4]   2> 1082434 T3933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1082434 T3933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1082434 T3933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1082435 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1082435 T3933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1082435 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1082435 T3933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1082435 T3933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1082436 T3933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1082437 T3933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1082437 T3933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1082438 T3933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1082438 T3933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50587/solr
   [junit4]   2> 1082438 T3933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1082439 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1082443 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082444 T3997 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082443 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082445 T4013 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6329e2d2 name:ZooKeeperConnection Watcher:127.0.0.1:50587 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1082444 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1082445 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1082447 T3933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1082449 T4015 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55f34a6e name:ZooKeeperConnection Watcher:127.0.0.1:50587/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1082449 T3933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1082453 T3933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1083454 T3933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50775_iw_w
   [junit4]   2> 1083456 T3933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50775_iw_w
   [junit4]   2> 1083459 T3957 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1083459 T3979 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1083459 T3966 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1083459 T4015 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1083459 T3997 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1083462 T4016 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1083462 T4016 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1083463 T4016 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1083463 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1083463 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1083463 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1083464 T3958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1083464 T3958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50775/iw_w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50775_iw_w",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1083464 T3958 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1083465 T3958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1083466 T3957 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1083567 T3957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1083567 T3997 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1083568 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1083567 T3966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1083568 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1084463 T4016 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1084464 T4016 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1084464 T4016 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1084464 T4016 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1084465 T4016 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1084465 T4016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/collection1/'
   [junit4]   2> 1084466 T4016 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/collection1/lib/.svn/' to classloader
   [junit4]   2> 1084466 T4016 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/collection1/lib/classes/' to classloader
   [junit4]   2> 1084466 T4016 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/collection1/lib/README' to classloader
   [junit4]   2> 1084482 T4016 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1084491 T4016 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1084492 T4016 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1084496 T4016 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1084592 T4016 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1084593 T4016 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1084594 T4016 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1084596 T4016 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1084606 T4016 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1084606 T4016 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1084606 T4016 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-005/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-67CA34973E01DF85-001/tempDir-001/jetty3/
   [junit4]   2> 1084606 T4016 oasc.JmxMonitoredMap.<init> J

[...truncated too long message...]

pse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  87) Thread[id=4283, name=searcherExecutor-2556-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  88) Thread[id=4284, name=Thread-1788, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:944)
   [junit4]   2> 	  89) Thread[id=4315, name=qtp1004922250-4315 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:81)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  90) Thread[id=4105, name=qtp873147827-4105 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  91) Thread[id=4333, name=coreZkRegister-2569-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  92) Thread[id=4451, name=searcherExecutor-2617-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=hi_IN, timezone=America/Iqaluit
   [junit4]   2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=112,free=200207024,total=461860864
   [junit4]   2> NOTE: All tests run in this JVM: [SpellingQueryConverterTest, TestCloudInspectUtil, TestStressLucene, LeaderElectionIntegrationTest, TestStressRecovery, TestPhraseSuggestions, TestSchemaResource, MultiThreadedOCPTest, SolrCoreCheckLockOnStartupTest, TestCSVResponseWriter, TestClassNameShortening, TestFieldTypeCollectionResource, TestDistributedSearch, TestSolrQueryParserResource, ZkNodePropsTest, TestInfoStreamLogging, SolrCmdDistributorTest, TestHighFrequencyDictionaryFactory, FullSolrCloudDistribCmdsTest, SortByFunctionTest, TestHighlightDedupGrouping, PluginInfoTest, SuggesterWFSTTest, DeleteReplicaTest, TestExpandComponent, TestReloadAndDeleteDocs, SignatureUpdateProcessorFactoryTest, TestJmxIntegration, TestSolrXmlPersistence, MultiTermTest, TestCollectionAPI, TestSolrDeletionPolicy2, TestFuzzyAnalyzedSuggestions, StatsComponentTest, SpellCheckCollatorTest, TestUpdate, ReplicationFactorTest, DistributedDebugComponentTest, TestManagedSynonymFilterFactory, TermVectorComponentDistributedTest, DistribDocExpirationUpdateProcessorTest, TestBinaryResponseWriter, TestUniqueKeyFieldResource, MinimalSchemaTest, RemoteQueryErrorTest, NumericFieldsTest, RAMDirectoryFactoryTest, PathHierarchyTokenizerFactoryTest, AlternateDirectoryTest, DateFieldTest, TestFiltering, HighlighterConfigTest, AtomicUpdatesTest, PreAnalyzedUpdateProcessorTest, TestFileDictionaryLookup, BadIndexSchemaTest, HdfsDirectoryTest, OutputWriterTest, TestQuerySenderNoQuery, UniqFieldsUpdateProcessorFactoryTest, SpellCheckComponentTest, DocumentAnalysisRequestHandlerTest, TestBinaryField, SuggesterTSTTest, XsltUpdateRequestHandlerTest, TestSimpleQParserPlugin, SimplePostToolTest, DefaultValueUpdateProcessorTest, IndexBasedSpellCheckerTest, TestXIncludeConfig, TestRangeQuery, CoreContainerCoreInitFailuresTest, HighlighterTest, HdfsSyncSliceTest, SolrXmlInZkTest, TestFreeTextSuggestions, ZkSolrClientTest, CollectionsAPIDistributedZkTest, TestReload, TestSchemaVersionResource, TestRTGBase, CurrencyFieldOpenExchangeTest, ShardRoutingCustomTest, TestSolrCoreProperties, ChaosMonkeyNothingIsSafeTest, TestDistributedGrouping, SolrPluginUtilsTest, FieldAnalysisRequestHandlerTest, TestHashPartitioner, TestComponentsName, DirectSolrConnectionTest, TimeZoneUtilsTest, HdfsCollectionsAPIDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=67CA34973E01DF85 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hi_IN -Dtests.timezone=America/Iqaluit -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([67CA34973E01DF85]:0)
   [junit4] Completed on J1 in 7224.86s, 1 test, 2 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 suite-level error, 1 error, 36 ignored (15 assumptions)

Total time: 173 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message