lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_20-ea-b15) - Build # 10447 - Failure!
Date Sun, 15 Jun 2014 11:08:49 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10447/
Java: 64bit/jdk1.8.0_20-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([10B46ED7F811B204]:0)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([10B46ED7F811B204]:0)




Build Log:
[...truncated 12110 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/init-core-data-001
   [junit4]   2> 240365 T1147 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 240366 T1147 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /oz/y
   [junit4]   2> 240370 T1147 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 240370 T1147 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 240371 T1148 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 240471 T1147 oasc.ZkTestServer.run start zk server on port:36016
   [junit4]   2> 240472 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 240523 T1154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ec0b003 name:ZooKeeperConnection Watcher:127.0.0.1:36016 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 240524 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 240524 T1147 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 240527 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 240528 T1156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d26d700 name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 240528 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 240529 T1147 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 240530 T1147 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 240532 T1147 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 240533 T1147 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 240534 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 240534 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 240537 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 240537 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 240540 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 240541 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 240542 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 240542 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 240543 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 240544 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 240545 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 240545 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 240547 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 240547 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 240548 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 240549 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 240550 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 240550 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 240551 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 240552 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 240553 T1147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 240553 T1147 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 240555 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 240556 T1158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79d45143 name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 240556 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 240776 T1147 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 240780 T1147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 240787 T1147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53779
   [junit4]   2> 240788 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 240788 T1147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 240788 T1147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002
   [junit4]   2> 240789 T1147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/'
   [junit4]   2> 240799 T1147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/solr.xml
   [junit4]   2> 240806 T1147 oasc.CoreContainer.<init> New CoreContainer 1412994380
   [junit4]   2> 240806 T1147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/]
   [junit4]   2> 240806 T1147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 240807 T1147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 240807 T1147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 240807 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 240807 T1147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 240807 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 240808 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 240808 T1147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 240808 T1147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 240810 T1147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 240810 T1147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 240810 T1147 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 240811 T1147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36016/solr
   [junit4]   2> 240811 T1147 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 240811 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 240813 T1169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5df58897 name:ZooKeeperConnection Watcher:127.0.0.1:36016 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 240814 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 240816 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 240817 T1171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb80760 name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 240817 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 240818 T1147 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 240820 T1147 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 240821 T1147 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 240823 T1147 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 240824 T1147 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 240826 T1147 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 240827 T1147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53779_oz%2Fy
   [junit4]   2> 240828 T1147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53779_oz%2Fy
   [junit4]   2> 240829 T1147 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 240830 T1147 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 240836 T1147 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53779_oz%2Fy
   [junit4]   2> 240836 T1147 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 240838 T1147 oasc.Overseer.start Overseer (id=91935425969455108-127.0.0.1:53779_oz%2Fy-n_0000000000) starting
   [junit4]   2> 240843 T1147 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 240849 T1173 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 240849 T1147 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 240851 T1147 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 240852 T1147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 240853 T1172 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 240855 T1174 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 240855 T1174 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 240856 T1174 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 240856 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 240857 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 240858 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53779/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53779_oz%2Fy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 240858 T1172 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 240858 T1172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 240859 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 240860 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 241856 T1174 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 241857 T1174 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 241858 T1174 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 241858 T1174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 241859 T1174 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 241860 T1174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/collection1/'
   [junit4]   2> 241860 T1174 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 241861 T1174 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 241861 T1174 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 241881 T1174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 241903 T1174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 241905 T1174 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 241911 T1174 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 241988 T1174 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 241989 T1174 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 241990 T1174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 241992 T1174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 242005 T1174 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 242005 T1174 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 242011 T1174 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data/
   [junit4]   2> 242012 T1174 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@22e17738
   [junit4]   2> 242012 T1174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data
   [junit4]   2> 242013 T1174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data/index/
   [junit4]   2> 242013 T1174 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 242015 T1174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data/index
   [junit4]   2> 242015 T1174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=86.06640625, floorSegmentMB=1.150390625, forceMergeDeletesPctAllowed=12.685136798340528, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 242019 T1174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 242019 T1174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 242021 T1174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 242021 T1174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 242021 T1174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 242021 T1174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 242022 T1174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 242022 T1174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 242022 T1174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 242022 T1174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 242023 T1174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 242023 T1174 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 242023 T1174 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 242024 T1174 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 242024 T1174 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 242024 T1174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 242024 T1174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 242025 T1174 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 242027 T1174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 242028 T1174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 242028 T1174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 242029 T1174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=24.126953125, floorSegmentMB=1.828125, forceMergeDeletesPctAllowed=25.269968079649903, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 242030 T1174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 242030 T1174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 242030 T1174 oass.SolrIndexSearcher.<init> Opening Searcher@7ab1b8f4[collection1] main
   [junit4]   2> 242030 T1174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 242031 T1174 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 242031 T1174 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 242032 T1174 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 242032 T1174 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 242032 T1174 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 242033 T1174 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 242033 T1174 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 242033 T1174 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 242033 T1174 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 242035 T1175 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ab1b8f4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 242035 T1174 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 242036 T1178 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53779/oz/y collection:control_collection shard:shard1
   [junit4]   2> 242036 T1147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 242037 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 242038 T1178 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 242039 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242040 T1180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f41f5be name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242041 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242041 T1147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 242042 T1178 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 242043 T1147 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 242043 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242043 T1178 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 242044 T1178 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C131 name=collection1 org.apache.solr.core.SolrCore@259a68ec url=https://127.0.0.1:53779/oz/y/collection1 node=127.0.0.1:53779_oz%2Fy C131_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:53779/oz/y, core=collection1, node_name=127.0.0.1:53779_oz%2Fy}
   [junit4]   2> 242044 T1178 C131 P53779 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53779/oz/y/collection1/
   [junit4]   2> 242044 T1178 C131 P53779 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 242044 T1178 C131 P53779 oasc.SyncStrategy.syncToMe https://127.0.0.1:53779/oz/y/collection1/ has no replicas
   [junit4]   2> 242044 T1178 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53779/oz/y/collection1/ shard1
   [junit4]   2> 242044 T1178 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 242045 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 242047 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242048 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242052 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242153 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 242153 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 242199 T1178 oasc.ZkController.register We are https://127.0.0.1:53779/oz/y/collection1/ and leader is https://127.0.0.1:53779/oz/y/collection1/
   [junit4]   2> 242199 T1178 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53779/oz/y
   [junit4]   2> 242199 T1178 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 242199 T1178 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 242199 T1178 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 242201 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242201 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242201 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242201 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 242202 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 242203 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53779/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53779_oz%2Fy",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 242204 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 242260 T1147 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 242261 T1147 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 242268 T1147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 242269 T1147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50635
   [junit4]   2> 242271 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 242272 T1147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 242272 T1147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003
   [junit4]   2> 242272 T1147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/'
   [junit4]   2> 242282 T1147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/solr.xml
   [junit4]   2> 242289 T1147 oasc.CoreContainer.<init> New CoreContainer 479543841
   [junit4]   2> 242289 T1147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/]
   [junit4]   2> 242290 T1147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 242290 T1147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 242290 T1147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 242290 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 242291 T1147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 242291 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 242291 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 242291 T1147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 242291 T1147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 242293 T1147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 242293 T1147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 242294 T1147 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 242294 T1147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36016/solr
   [junit4]   2> 242294 T1147 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 242295 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242296 T1191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@703c8297 name:ZooKeeperConnection Watcher:127.0.0.1:36016 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242296 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242298 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242299 T1193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb3087f name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242299 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242304 T1147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 242305 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 242305 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 242306 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 243306 T1147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50635_oz%2Fy
   [junit4]   2> 243308 T1147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50635_oz%2Fy
   [junit4]   2> 243310 T1193 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 243310 T1180 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 243310 T1171 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 243317 T1194 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 243317 T1194 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 243318 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 243318 T1194 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 243319 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 243319 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 243321 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 243322 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50635/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50635_oz%2Fy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 243322 T1172 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 243323 T1172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 243326 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 243429 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 243429 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 243429 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244319 T1194 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 244320 T1194 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 244320 T1194 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 244321 T1194 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 244322 T1194 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 244322 T1194 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/collection1/'
   [junit4]   2> 244323 T1194 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 244324 T1194 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 244324 T1194 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 244348 T1194 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 244367 T1194 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 244369 T1194 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 244374 T1194 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 244453 T1194 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 244454 T1194 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 244455 T1194 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 244457 T1194 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 244468 T1194 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 244469 T1194 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 244469 T1194 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1/
   [junit4]   2> 244469 T1194 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@22e17738
   [junit4]   2> 244469 T1194 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1
   [junit4]   2> 244470 T1194 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1/index/
   [junit4]   2> 244470 T1194 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 244471 T1194 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1/index
   [junit4]   2> 244471 T1194 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=86.06640625, floorSegmentMB=1.150390625, forceMergeDeletesPctAllowed=12.685136798340528, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 244475 T1194 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 244476 T1194 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 244478 T1194 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 244478 T1194 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 244479 T1194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 244479 T1194 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 244479 T1194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 244479 T1194 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 244479 T1194 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 244480 T1194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 244480 T1194 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 244480 T1194 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 244480 T1194 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 244481 T1194 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 244481 T1194 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 244481 T1194 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 244481 T1194 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 244482 T1194 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 244483 T1194 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 244484 T1194 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 244484 T1194 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 244485 T1194 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=24.126953125, floorSegmentMB=1.828125, forceMergeDeletesPctAllowed=25.269968079649903, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 244487 T1194 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 244487 T1194 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 244488 T1194 oass.SolrIndexSearcher.<init> Opening Searcher@7a190835[collection1] main
   [junit4]   2> 244488 T1194 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 244489 T1194 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 244490 T1194 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 244490 T1194 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 244490 T1194 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 244491 T1194 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 244492 T1194 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 244492 T1194 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 244492 T1194 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 244492 T1194 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 244496 T1195 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a190835[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 244499 T1194 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 244500 T1198 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50635/oz/y collection:collection1 shard:shard2
   [junit4]   2> 244500 T1147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 244500 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 244501 T1198 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 244505 T1198 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 244507 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244507 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244507 T1198 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 244507 T1198 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 244507 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C132 name=collection1 org.apache.solr.core.SolrCore@45637686 url=https://127.0.0.1:50635/oz/y/collection1 node=127.0.0.1:50635_oz%2Fy C132_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50635/oz/y, core=collection1, node_name=127.0.0.1:50635_oz%2Fy}
   [junit4]   2> 244507 T1198 C132 P50635 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50635/oz/y/collection1/
   [junit4]   2> 244507 T1198 C132 P50635 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 244508 T1198 C132 P50635 oasc.SyncStrategy.syncToMe https://127.0.0.1:50635/oz/y/collection1/ has no replicas
   [junit4]   2> 244508 T1198 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50635/oz/y/collection1/ shard2
   [junit4]   2> 244508 T1198 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 244508 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 244510 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244510 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244512 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244615 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244615 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244615 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244661 T1198 oasc.ZkController.register We are https://127.0.0.1:50635/oz/y/collection1/ and leader is https://127.0.0.1:50635/oz/y/collection1/
   [junit4]   2> 244661 T1198 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50635/oz/y
   [junit4]   2> 244662 T1198 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 244662 T1198 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 244662 T1198 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 244664 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244664 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244664 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244666 T1198 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 244667 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 244668 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50635/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50635_oz%2Fy",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 244669 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 244716 T1147 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 244717 T1147 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 244721 T1147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 244722 T1147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:41225
   [junit4]   2> 244723 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 244724 T1147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 244724 T1147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004
   [junit4]   2> 244724 T1147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/'
   [junit4]   2> 244735 T1147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/solr.xml
   [junit4]   2> 244742 T1147 oasc.CoreContainer.<init> New CoreContainer 423296912
   [junit4]   2> 244742 T1147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/]
   [junit4]   2> 244742 T1147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 244743 T1147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 244743 T1147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 244743 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 244743 T1147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 244744 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 244744 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 244744 T1147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 244744 T1147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 244746 T1147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 244746 T1147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 244747 T1147 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 244747 T1147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36016/solr
   [junit4]   2> 244747 T1147 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 244748 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 244749 T1209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ec167e1 name:ZooKeeperConnection Watcher:127.0.0.1:36016 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 244750 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 244751 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 244752 T1211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d1266b5 name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 244752 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 244756 T1147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 244771 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244771 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244771 T1211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 244771 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 245758 T1147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41225_oz%2Fy
   [junit4]   2> 245760 T1147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41225_oz%2Fy
   [junit4]   2> 245762 T1193 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245762 T1211 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245762 T1171 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245762 T1180 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245769 T1212 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 245769 T1212 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 245770 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 245770 T1212 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 245770 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 245770 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 245772 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 245772 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:41225/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41225_oz%2Fy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 245772 T1172 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 245772 T1172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 245774 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 245876 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 245876 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 245876 T1211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 245876 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 246770 T1212 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 246771 T1212 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 246771 T1212 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 246772 T1212 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 246772 T1212 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 246772 T1212 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/collection1/'
   [junit4]   2> 246773 T1212 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 246774 T1212 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 246774 T1212 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 246804 T1212 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 246812 T1212 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 246813 T1212 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 246817 T1212 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 246895 T1212 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 246896 T1212 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 246897 T1212 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 246899 T1212 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 246913 T1212 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 246913 T1212 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 246913 T1212 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2/
   [junit4]   2> 246913 T1212 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@22e17738
   [junit4]   2> 246914 T1212 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2
   [junit4]   2> 246914 T1212 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2/index/
   [junit4]   2> 246915 T1212 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 246915 T1212 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2/index
   [junit4]   2> 246918 T1212 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=86.06640625, floorSegmentMB=1.150390625, forceMergeDeletesPctAllowed=12.685136798340528, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 247621 T1212 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 247621 T1212 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 247623 T1212 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 247624 T1212 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 247624 T1212 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 247624 T1212 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 247624 T1212 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 247624 T1212 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 247625 T1212 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 247625 T1212 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 247625 T1212 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 247625 T1212 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 247626 T1212 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 247626 T1212 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 247626 T1212 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 247626 T1212 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 247626 T1212 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 247627 T1212 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 247629 T1212 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 247630 T1212 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 247630 T1212 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 247631 T1212 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=24.126953125, floorSegmentMB=1.828125, forceMergeDeletesPctAllowed=25.269968079649903, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 247632 T1212 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 247632 T1212 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 247633 T1212 oass.SolrIndexSearcher.<init> Opening Searcher@54f42ee3[collection1] main
   [junit4]   2> 247633 T1212 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 247634 T1212 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 247634 T1212 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 247634 T1212 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 247634 T1212 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 247635 T1212 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 247635 T1212 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 247635 T1212 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 247635 T1212 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 247635 T1212 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 247638 T1213 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@54f42ee3[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 247639 T1212 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 247640 T1216 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:41225/oz/y collection:collection1 shard:shard3
   [junit4]   2> 247640 T1147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 247641 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 247641 T1216 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 247644 T1216 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 247646 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247646 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247646 T1216 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 247646 T1216 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C133 name=collection1 org.apache.solr.core.SolrCore@671bb0ae url=https://127.0.0.1:41225/oz/y/collection1 node=127.0.0.1:41225_oz%2Fy C133_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:41225/oz/y, core=collection1, node_name=127.0.0.1:41225_oz%2Fy}
   [junit4]   2> 247646 T1216 C133 P41225 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:41225/oz/y/collection1/
   [junit4]   2> 247646 T1216 C133 P41225 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 247647 T1216 C133 P41225 oasc.SyncStrategy.syncToMe https://127.0.0.1:41225/oz/y/collection1/ has no replicas
   [junit4]   2> 247647 T1216 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:41225/oz/y/collection1/ shard3
   [junit4]   2> 247647 T1216 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 247646 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247649 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247650 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 247652 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247652 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247652 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247652 T1211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247655 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 247656 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247758 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247758 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247758 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247758 T1211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247802 T1216 oasc.ZkController.register We are https://127.0.0.1:41225/oz/y/collection1/ and leader is https://127.0.0.1:41225/oz/y/collection1/
   [junit4]   2> 247802 T1216 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:41225/oz/y
   [junit4]   2> 247802 T1216 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 247803 T1216 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 247803 T1216 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 247804 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247804 T1216 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 247805 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247805 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247806 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 247806 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:41225/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41225_oz%2Fy",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 247814 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 247909 T1147 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 247910 T1147 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 247913 T1147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 247914 T1147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40640
   [junit4]   2> 247916 T1147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 247916 T1147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 247916 T1147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005
   [junit4]   2> 247916 T1147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/'
   [junit4]   2> 247918 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247918 T1211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247918 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247918 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 247928 T1147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/solr.xml
   [junit4]   2> 247936 T1147 oasc.CoreContainer.<init> New CoreContainer 2062668296
   [junit4]   2> 247936 T1147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/]
   [junit4]   2> 247936 T1147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 247937 T1147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 247937 T1147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 247937 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 247937 T1147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 247938 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 247938 T1147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 247938 T1147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 247938 T1147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 247940 T1147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 247940 T1147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 247940 T1147 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 247941 T1147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36016/solr
   [junit4]   2> 247941 T1147 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 247942 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 247943 T1227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f490631 name:ZooKeeperConnection Watcher:127.0.0.1:36016 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 247943 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 247945 T1147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 247951 T1229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@211f2221 name:ZooKeeperConnection Watcher:127.0.0.1:36016/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 247951 T1147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 247957 T1147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 248959 T1147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40640_oz%2Fy
   [junit4]   2> 248960 T1147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40640_oz%2Fy
   [junit4]   2> 248962 T1180 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 248963 T1229 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 248963 T1171 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 248963 T1193 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 248962 T1211 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 248967 T1230 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 248967 T1230 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 248968 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 248968 T1230 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 248968 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 248968 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 248969 T1172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 248969 T1172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40640/oz/y",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40640_oz%2Fy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 248969 T1172 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 248969 T1172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 248971 T1171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 249139 T1211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 249140 T1229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 249140 T1193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 249140 T1171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 249139 T1180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 249968 T1230 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 249968 T1230 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 249969 T1230 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 249969 T1230 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 249970 T1230 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 249970 T1230 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/collection1/'
   [junit4]   2> 249971 T1230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/collection1/lib/.svn/' to classloader
   [junit4]   2> 249971 T1230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/collection1/lib/classes/' to classloader
   [junit4]   2> 249971 T1230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-10B46ED7F811B204-001/tempDir-005/collection1/lib/README' to classloader
   [junit4]   2> 250009 T1230 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 250020 T1230 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 250021 T1230 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 250028 T1230 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 250148 T1230 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 250150 T1230 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 250151 T1230 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2>

[...truncated too long message...]

1 SslSelectChannelConnector@127.0.0.1:40433, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  85) Thread[id=1342, name=qtp1783659049-1342 Acceptor1 SslSelectChannelConnector@127.0.0.1:54471, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  86) Thread[id=1610, name=HashSessionScavenger-67, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  87) Thread[id=1500, name=Thread-731-SendThread(127.0.0.1:36016), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 	  88) Thread[id=1491, name=qtp953088093-1491 Acceptor0 SslSelectChannelConnector@127.0.0.1:51875, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  89) Thread[id=1513, name=HashSessionScavenger-63, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  90) Thread[id=1236, name=qtp708036054-1236 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  91) Thread[id=1588, name=HashSessionScavenger-66, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  92) Thread[id=1228, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[10B46ED7F811B204]-SendThread(127.0.0.1:36016), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 7466969 T1246 oaz.ClientCnxn$SendThread.run WARN Session 0x1469ec8672f000d for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 7466995 T1500 oaz.ClientCnxn$SendThread.run WARN Session 0x1469ec8672f0023 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 7467024 T1349 oaz.ClientCnxn$SendThread.run WARN Session 0x1469ec8672f0017 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene49: {rnd_b=PostingsFormat(name=Memory doPackFST= false), _version_=MockVariableIntBlock(baseBlockSize=62), a_t=MockVariableIntBlock(baseBlockSize=62), a_si=FST41, id=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=DefaultSimilarity, locale=in_ID, timezone=Australia/Lord_Howe
   [junit4]   2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_20-ea (64-bit)/cpus=8,threads=112,free=204710240,total=457179136
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, RAMDirectoryFactoryTest, TestCloudManagedSchema, TestOmitPositions, TestDefaultSimilarityFactory, TestXIncludeConfig, CurrencyFieldXmlFileTest, BasicDistributedZk2Test, UniqFieldsUpdateProcessorFactoryTest, TestMiniSolrCloudCluster, PreAnalyzedFieldTest, SSLMigrationTest, MergeStrategyTest, TestBinaryField, TestImplicitCoreProperties, RequestHandlersTest, TestSolrDeletionPolicy2, AsyncMigrateRouteKeyTest, HdfsUnloadDistributedZkTest, TestMaxScoreQueryParser, TestFastWriter, SolrCmdDistributorTest, TestFastOutputStream, JSONWriterTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=10B46ED7F811B204 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in_ID -Dtests.timezone=Australia/Lord_Howe -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([10B46ED7F811B204]:0)
   [junit4] Completed on J1 in 7226.79s, 1 test, 2 errors <<< FAILURES!

[...truncated 245 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 405 suites, 1654 tests, 1 suite-level error, 1 error, 332 ignored (16 assumptions)

Total time: 161 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_20-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message