lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Rowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-12709) Simulate a 1 bln docs scaling-up scenario
Date Mon, 01 Oct 2018 18:53:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-12709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634477#comment-16634477 ] 

Steve Rowe commented on SOLR-12709:
-----------------------------------

{{TestSimExtremeIndexing.testScaleUp()}} failure from [https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1135/], the seed only reproduced 1/5 iterations for me locally:

{noformat}
Checking out Revision 2369c8963412773592098475bdd8af1da81e3ac5 (refs/remotes/origin/master)
[...]
   [smoker]    [junit4]   2> 1221947 DEBUG (TEST-TestSimExtremeIndexing.testScaleUp-seed#[4D17B79C7402215C]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 218
   [smoker]    [junit4]   2> 1222001 INFO  (TEST-TestSimExtremeIndexing.testScaleUp-seed#[4D17B79C7402215C]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [smoker]    [junit4]   2> ############ CLUSTER STATE ############
   [smoker]    [junit4]   2> #######################################
   [smoker]    [junit4]   2> ## Live nodes:		200
   [smoker]    [junit4]   2> ## Empty nodes:	191
   [smoker]    [junit4]   2> ## Dead nodes:		0
   [smoker]    [junit4]   2> ## Collections:
   [smoker]    [junit4]   2> ##  * .system
   [smoker]    [junit4]   2> ##    shardsTotal	1
   [smoker]    [junit4]   2> ##    shardsState	{active=1}
   [smoker]    [junit4]   2> ##      shardsWithoutLeader	0
   [smoker]    [junit4]   2> ##    totalReplicas	1
   [smoker]    [junit4]   2> ##      activeReplicas	1
   [smoker]    [junit4]   2> ##      inactiveReplicas	0
   [smoker]    [junit4]   2> ##    totalActiveDocs	34
   [smoker]    [junit4]   2> ##      bufferedDocs	0
   [smoker]    [junit4]   2> ##      maxActiveSliceDocs	34
   [smoker]    [junit4]   2> ##      minActiveSliceDocs	34
   [smoker]    [junit4]   2> ##      avgActiveSliceDocs	34
   [smoker]    [junit4]   2> ##    totalInactiveDocs	0
   [smoker]    [junit4]   2> ##      maxInactiveSliceDocs	0
   [smoker]    [junit4]   2> ##      minInactiveSliceDocs	0
   [smoker]    [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [smoker]    [junit4]   2> ##    totalActiveBytes	27,240
   [smoker]    [junit4]   2> ##      maxActiveSliceBytes	27,240
   [smoker]    [junit4]   2> ##      minActiveSliceBytes	27,240
   [smoker]    [junit4]   2> ##      avgActiveSliceBytes	27,240
   [smoker]    [junit4]   2> ##    totalInactiveBytes	0
   [smoker]    [junit4]   2> ##      maxInactiveSliceBytes	0
   [smoker]    [junit4]   2> ##      minInactiveSliceBytes	0
   [smoker]    [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [smoker]    [junit4]   2> ##    totalActiveDeletedDocs	0
   [smoker]    [junit4]   2> ##  * testScaleUp_collection
   [smoker]    [junit4]   2> ##    shardsTotal	92
   [smoker]    [junit4]   2> ##    shardsState	{active=46, construction=2, inactive=44}
   [smoker]    [junit4]   2> ##      shardsWithoutLeader	0
   [smoker]    [junit4]   2> ##    totalReplicas	184
   [smoker]    [junit4]   2> ##      activeReplicas	92
   [smoker]    [junit4]   2> ##      inactiveReplicas	92
   [smoker]    [junit4]   2> ##    totalActiveDocs	1,000,000,000
   [smoker]    [junit4]   2> ##      bufferedDocs	204,400,000
   [smoker]    [junit4]   2> ##      maxActiveSliceDocs	45,600,000
   [smoker]    [junit4]   2> ##      minActiveSliceDocs	15,625,000
   [smoker]    [junit4]   2> ##      avgActiveSliceDocs	17,295,652
   [smoker]    [junit4]   2> ##    totalInactiveDocs	1,634,193,750
   [smoker]    [junit4]   2> ##      maxInactiveSliceDocs	58,650,000
   [smoker]    [junit4]   2> ##      minInactiveSliceDocs	22,800,000
   [smoker]    [junit4]   2> ##      avgInactiveSliceDocs	35,525,951
   [smoker]    [junit4]   2> ##    totalActiveBytes	390,916,321,320
   [smoker]    [junit4]   2> ##      maxActiveSliceBytes	19,875,010,240
   [smoker]    [junit4]   2> ##      minActiveSliceBytes	7,673,447,740
   [smoker]    [junit4]   2> ##      avgActiveSliceBytes	8,498,180,898
   [smoker]    [junit4]   2> ##    totalInactiveBytes	787,600,706,560
   [smoker]    [junit4]   2> ##      maxInactiveSliceBytes	28,125,010,240
   [smoker]    [junit4]   2> ##      minInactiveSliceBytes	11,400,010,240
   [smoker]    [junit4]   2> ##      avgInactiveSliceBytes	17,121,754,490
   [smoker]    [junit4]   2> ##    totalActiveDeletedDocs	0
   [smoker]    [junit4]   2> ## Max replicas per node:	45
   [smoker]    [junit4]   2> ## Min replicas per node:	0
   [smoker]    [junit4]   2> ## Total replicas:		185
   [smoker]    [junit4]   2> ## * .system		1
   [smoker]    [junit4]   2> ##		- active           1
   [smoker]    [junit4]   2> ## * testScaleUp_collection		184
   [smoker]    [junit4]   2> ##		- active         184
   [smoker]    [junit4]   2> ######### Solr op counts ##########
   [smoker]    [junit4]   2> ##		- CREATE             1
   [smoker]    [junit4]   2> ##		- DELETESTATUS      51
   [smoker]    [junit4]   2> ##		- REQUESTSTATUS     51
   [smoker]    [junit4]   2> ##		- SPLITSHARD        52
   [smoker]    [junit4]   2> ##		- autoscaling        1
   [smoker]    [junit4]   2> ##		- query              1
   [smoker]    [junit4]   2> ##		- update          5042
   [smoker]    [junit4]   2> ######### Autoscaling event counts ###########
   [smoker]    [junit4]   2> ## * Trigger: scaleUpTrigger
   [smoker]    [junit4]   2> ##		- FAILED          1
   [smoker]    [junit4]   2> ##		- STARTED         8
   [smoker]    [junit4]   2> ##		- SUCCEEDED       1
   [smoker]    [junit4]   2> 
   [smoker]    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimExtremeIndexing -Dtests.method=testScaleUp -Dtests.seed=4D17B79C7402215C -Dtests.multiplier=2 -Dtests.locale=ro -Dtests.timezone=America/Catamarca -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [smoker]    [junit4] FAILURE 4.88s J1 | TestSimExtremeIndexing.testScaleUp <<<
   [smoker]    [junit4]    > Throwable #1: java.lang.AssertionError: {numFound=795600000,start=0,docs=[]} expected:<1000000000> but was:<795600000>
   [smoker]    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4D17B79C7402215C:6C49F13E782CFFFD]:0)
   [smoker]    [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimExtremeIndexing.testScaleUp(TestSimExtremeIndexing.java:121)
   [smoker]    [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [smoker]    [junit4]   2> 1222003 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [smoker]    [junit4]   2> 1222003 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [smoker]    [junit4]   2> 1222003 WARN  (SolrRrdBackendFactory-5969-thread-1) [    ] o.a.s.m.r.SolrRrdBackendFactory Error updating RRD data for solr.jvm
   [smoker]    [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [smoker]    [junit4]   2> Caused by: java.lang.InterruptedException
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
   [smoker]    [junit4]   2> 	... 15 more
   [smoker]    [junit4]   2> 1222004 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [smoker]    [junit4]   2> 1222004 DEBUG (simCloudManagerPool-5971-thread-33) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
   [smoker]    [junit4]   2> 1222005 WARN  (SolrRrdBackendFactory-5969-thread-1) [    ] o.a.s.m.r.SolrRrdBackendFactory Error updating RRD data for solr.collection.testScaleUp_collection
   [smoker]    [junit4]   2> java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [smoker]    [junit4]   2> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
   [smoker]    [junit4]   2> 	... 15 more
   [smoker]    [junit4]   2> Caused by: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650) ~[test/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [smoker]    [junit4]   2> 	... 3 more
   [smoker]    [junit4]   2> 1222006 WARN  (AutoscalingActionExecutor-5973-thread-1) [    ] o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [smoker]    [junit4]   2> java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:792) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:310) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [smoker]    [junit4]   2> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
   [smoker]    [junit4]   2> 	... 9 more
   [smoker]    [junit4]   2> Caused by: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650) ~[test/:?]
   [smoker]    [junit4]   2> 	... 5 more
   [smoker]    [junit4]   2> 1222009 DEBUG (simCloudManagerPool-5971-thread-33) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 219
   [smoker]    [junit4]   2> 1222009 WARN  (AutoscalingActionExecutor-5973-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [smoker]    [junit4]   2> org.apache.solr.cloud.autoscaling.TriggerActionException: Error processing action for trigger event: {
   [smoker]    [junit4]   2>   "id":"1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp",
   [smoker]    [junit4]   2>   "source":"scaleUpTrigger",
   [smoker]    [junit4]   2>   "eventTime":5366401710726204,
   [smoker]    [junit4]   2>   "eventType":"INDEXSIZE",
   [smoker]    [junit4]   2>   "properties":{
   [smoker]    [junit4]   2>     "__start__":25,
   [smoker]    [junit4]   2>     "aboveSize":{
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_0_0_replica_n217":"docs=24712500, bytes=12075020480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_0_1_replica_n223":"docs=24712500, bytes=12075010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_1_0_replica_n127":"docs=24712500, bytes=12015645480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_1_1_replica_n133":"docs=24712500, bytes=12015635240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_0_0_replica_n199":"docs=24712500, bytes=12065645480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_0_1_replica_n203":"docs=24712500, bytes=12065635240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_1_0_replica_n179":"docs=24712500, bytes=12103145480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_1_1_replica_n183":"docs=24712500, bytes=12103135240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_1_replica_n11":"docs=45600000, bytes=19875010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_0_0_replica_n137":"docs=24712500, bytes=11990645480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_0_1_replica_n143":"docs=24712500, bytes=11990635240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_1_0_replica_n237":"docs=24712500, bytes=12075020480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_1_1_replica_n243":"docs=24712500, bytes=12075010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_0_0_replica_n227":"docs=24712500, bytes=12081270480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_0_1_replica_n233":"docs=24712500, bytes=12081260240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_1_0_replica_n167":"docs=24712500, bytes=12053145480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_1_1_replica_n173":"docs=24712500, bytes=12053135240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_0_0_replica_n147":"docs=24712500, bytes=12003145480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_0_1_replica_n153":"docs=24712500, bytes=12003135240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_1_0_replica_n209":"docs=24712500, bytes=12156270480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_1_1_replica_n213":"docs=24712500, bytes=12156260240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_0_0_replica_n189":"docs=24712500, bytes=12200020480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_0_1_replica_n193":"docs=24712500, bytes=12200010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_1_0_replica_n157":"docs=24712500, bytes=12068770480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_1_1_replica_n163":"docs=24712500, bytes=12068760240"},
   [smoker]    [junit4]   2>     "belowSize":{},
   [smoker]    [junit4]   2>     "_enqueue_time_":5366407403469004,
   [smoker]    [junit4]   2>     "requestedOps":[
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_1_1"}]}}]}}
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:311) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [smoker]    [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception while processing event: {
   [smoker]    [junit4]   2>   "id":"1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp",
   [smoker]    [junit4]   2>   "source":"scaleUpTrigger",
   [smoker]    [junit4]   2>   "eventTime":5366401710726204,
   [smoker]    [junit4]   2>   "eventType":"INDEXSIZE",
   [smoker]    [junit4]   2>   "properties":{
   [smoker]    [junit4]   2>     "__start__":25,
   [smoker]    [junit4]   2>     "aboveSize":{
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_0_0_replica_n217":"docs=24712500, bytes=12075020480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_0_1_replica_n223":"docs=24712500, bytes=12075010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_1_0_replica_n127":"docs=24712500, bytes=12015645480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_0_1_1_replica_n133":"docs=24712500, bytes=12015635240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_0_0_replica_n199":"docs=24712500, bytes=12065645480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_0_1_replica_n203":"docs=24712500, bytes=12065635240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_1_0_replica_n179":"docs=24712500, bytes=12103145480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_0_1_1_1_replica_n183":"docs=24712500, bytes=12103135240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard1_1_replica_n11":"docs=45600000, bytes=19875010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_0_0_replica_n137":"docs=24712500, bytes=11990645480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_0_1_replica_n143":"docs=24712500, bytes=11990635240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_1_0_replica_n237":"docs=24712500, bytes=12075020480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_0_1_1_replica_n243":"docs=24712500, bytes=12075010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_0_0_replica_n227":"docs=24712500, bytes=12081270480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_0_1_replica_n233":"docs=24712500, bytes=12081260240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_1_0_replica_n167":"docs=24712500, bytes=12053145480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_0_1_1_1_replica_n173":"docs=24712500, bytes=12053135240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_0_0_replica_n147":"docs=24712500, bytes=12003145480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_0_1_replica_n153":"docs=24712500, bytes=12003135240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_1_0_replica_n209":"docs=24712500, bytes=12156270480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_0_1_1_replica_n213":"docs=24712500, bytes=12156260240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_0_0_replica_n189":"docs=24712500, bytes=12200020480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_0_1_replica_n193":"docs=24712500, bytes=12200010240",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_1_0_replica_n157":"docs=24712500, bytes=12068770480",
   [smoker]    [junit4]   2>       "testScaleUp_collection_shard2_1_1_1_1_replica_n163":"docs=24712500, bytes=12068760240"},
   [smoker]    [junit4]   2>     "belowSize":{},
   [smoker]    [junit4]   2>     "_enqueue_time_":5366407403469004,
   [smoker]    [junit4]   2>     "requestedOps":[
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_0_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_0_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_1_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_1_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_0_1_0_1"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard1_0_0_1_0"}]}},
   [smoker]    [junit4]   2>       {
   [smoker]    [junit4]   2>         "action":"SPLITSHARD",
   [smoker]    [junit4]   2>         "hints":{"COLL_SHARD":[{
   [smoker]    [junit4]   2>               "first":"testScaleUp_collection",
   [smoker]    [junit4]   2>               "second":"shard2_1_0_1_1"}]}}]}}
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:120) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
   [smoker]    [junit4]   2> 	... 6 more
   [smoker]    [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception executing operation: action=SPLITSHARD&async=scaleUpTrigger/1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp/0&waitForFinalState=true&collection=testScaleUp_collection&shard=shard2_0_1_0_1
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:110) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
   [smoker]    [junit4]   2> 	... 6 more
   [smoker]    [junit4]   2> Caused by: java.io.IOException: java.lang.InterruptedException
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
   [smoker]    [junit4]   2> 	... 6 more
   [smoker]    [junit4]   2> Caused by: java.lang.InterruptedException
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
   [smoker]    [junit4]   2> 	... 6 more
   [smoker]    [junit4]   2> 1222010 DEBUG (AutoscalingActionExecutor-5973-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 176888 ms for event id=1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp
   [smoker]    [junit4]   2> 1222007 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [smoker]    [junit4]   2> 1222010 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [smoker]    [junit4]   2> 1222010 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [smoker]    [junit4]   2> 1222011 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [smoker]    [junit4]   2> 1222011 WARN  (SolrRrdBackendFactory-5969-thread-1) [    ] o.a.s.m.r.SolrRrdBackendFactory Error updating RRD data for solr.node
   [smoker]    [junit4]   2> java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [smoker]    [junit4]   2> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
   [smoker]    [junit4]   2> 	... 15 more
   [smoker]    [junit4]   2> Caused by: java.io.IOException: already closed
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650) ~[test/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [smoker]    [junit4]   2> 	... 3 more
   [smoker]    [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-8.0.0/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimExtremeIndexing_4D17B79C7402215C-001
   [smoker]    [junit4]   2> 1222029 WARN  (SolrRrdBackendFactory-5969-thread-1) [    ] o.a.s.m.r.SolrRrdBackendFactory Error sending RRD data updates
   [smoker]    [junit4]   2> java.io.IOException: java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$129/198524200@1817d8df rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2a463d6b[Shutting down, pool size = 142, active threads = 0, queued tasks = 0, completed tasks = 5300]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:448) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:366) ~[java/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [smoker]    [junit4]   2> Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$129/198524200@1817d8df rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2a463d6b[Shutting down, pool size = 142, active threads = 0, queued tasks = 0, completed tasks = 5300]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194) ~[java/:?]
   [smoker]    [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_172]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.submit(SimCloudManager.java:569) ~[test/:?]
   [smoker]    [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:650) ~[test/:?]
   [smoker]    [junit4]   2> 	... 13 more
   [smoker]    [junit4]   2> Sep 27, 2018 7:42:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [smoker]    [junit4]   2> WARNING: Will linger awaiting termination of 18 leaked thread(s).
   [smoker]    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2a3300f8), locale=ro, timezone=America/Catamarca
   [smoker]    [junit4]   2> NOTE: Linux 4.4.0-130-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=223113456,total=518520832
   [smoker]    [junit4]   2> NOTE: All tests run in this JVM: [SearchHandlerTest, AutoScalingHandlerTest, TestRangeQuery, TestPerFieldSimilarity, TestBlobHandler, TestManagedSchema, ParsingFieldUpdateProcessorsTest, MetricsHistoryHandlerTest, TestLegacyTerms, DirectSolrSpellCheckerTest, ConfigSetsAPITest, TestConfig, TestNumericRangeQuery64, TestQueryUtils, TestValueSourceCache, SolrInfoBeanTest, TestMiniSolrCloudClusterSSL, SolrIndexSplitterTest, ProtectedTermFilterFactoryTest, DefaultValueUpdateProcessorTest, TestClusterProperties, TestAuthorizationFramework, TriggerSetPropertiesIntegrationTest, TestMacroExpander, TestSSLRandomization, TestDistribDocBasedVersion, FacetPivotSmallTest, HLLUtilTest, CdcrRequestHandlerTest, TestWithCollection, DateRangeFieldTest, V2StandaloneTest, TestCoreDiscovery, BasicFunctionalityTest, TestSolrIndexConfig, CachingDirectoryFactoryTest, NodeMarkersRegistrationTest, TestReload, DeleteStatusTest, DisMaxRequestHandlerTest, TestJavabinTupleStreamParser, TestManagedResource, TestCustomDocTransformer, TestCloudJSONFacetSKG, AnalysisErrorHandlingTest, JavabinLoaderTest, TestReversedWildcardFilterFactory, TestSimNodeAddedTrigger, SimpleMLTQParserTest, HealthCheckHandlerTest, JvmMetricsTest, TestBlendedInfixSuggestions, TestLMDirichletSimilarityFactory, TestFaceting, TestDirectoryFactory, HdfsRecoverLeaseTest, TestExceedMaxTermLength, TestDynamicFieldResource, TestNRTOpen, TestHdfsCloudBackupRestore, SpellCheckCollatorTest, TestUpdate, TestTolerantSearch, UtilsToolTest, TestFastLRUCache, BasicAuthIntegrationTest, TestOmitPositions, QueryElevationComponentTest, TestDistributedSearch, TestPayloadScoreQParserPlugin, PeerSyncTest, SyncSliceTest, HdfsDirectoryTest, ActionThrottleTest, DistributedSpellCheckComponentTest, HdfsChaosMonkeyNothingIsSafeTest, TriggerCooldownIntegrationTest, LukeRequestHandlerTest, TestNumericTerms64, TestImplicitCoreProperties, LargeFieldTest, TestDistributedStatsComponentCardinality, DistributedQueryComponentCustomSortTest, TestJsonFacetRefinement, WrapperMergePolicyFactoryTest, CdcrBootstrapTest, TestExclusionRuleCollectionAccess, PeerSyncReplicationTest, BasicDistributedZkTest, ImplicitSnitchTest, DirectUpdateHandlerTest, TestClusterStateMutator, CollectionsAPIAsyncDistributedZkTest, LeaderFailureAfterFreshStartTest, ScriptEngineTest, DistributedFacetSimpleRefinementLongTailTest, SimplePostToolTest, TestSolr4Spatial, SuggesterFSTTest, SchemaApiFailureTest, TestNoOpRegenerator, DirectoryFactoryTest, TestBulkSchemaConcurrent, ChaosMonkeyNothingIsSafeWithPullReplicasTest, OpenCloseCoreStressTest, ReplicationFactorTest, TestSolrCloudWithHadoopAuthPlugin, TestReplicationHandlerBackup, TestUseDocValuesAsStored2, DistributedDebugComponentTest, TestTestInjection, MultiTermTest, ZkNodePropsTest, NodeAddedTriggerTest, Tagger2Test, DocValuesNotIndexedTest, TestMinMaxOnMultiValuedField, QueryParsingTest, TestSolrQueryResponse, FieldMutatingUpdateProcessorTest, AssignBackwardCompatibilityTest, TermVectorComponentDistributedTest, TestFieldTypeCollectionResource, TestUnifiedSolrHighlighter, StatsComponentTest, TestSort, TestCollationField, TestConfigSetsAPIZkFailure, TestSimDistribStateManager, PreAnalyzedFieldTest, BadComponentTest, NumberUtilsTest, SpatialRPTFieldTypeTest, TestSolrCloudSnapshots, DeleteInactiveReplicaTest, DeleteNodeTest, DistribJoinFromCollectionTest, DistributedQueueTest, DistributedVersionInfoTest, LIROnShardRestartTest, LIRRollingUpdatesTest, MoveReplicaTest, OverseerStatusTest, RecoveryZkTest, RemoteQueryErrorTest, ReplaceNodeNoTargetTest, SSLMigrationTest, SaslZkACLProviderTest, ShardRoutingTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestCloudConsistency, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestDeleteCollectionOnDownNodes, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestPullReplica, TestPullReplicaErrorHandling, CollectionReloadTest, CustomCollectionTest, AutoAddReplicasPlanActionTest, ExecutePlanActionTest, MetricTriggerIntegrationTest, ScheduledTriggerTest, SearchRateTriggerIntegrationTest, SystemLogListenerTest, TestSimExtremeIndexing]
   [smoker]    [junit4] Completed [539/835 (1!)] on J1 in 5.23s, 1 test, 1 failure <<< FAILURES!
{noformat}

> Simulate a 1 bln docs scaling-up scenario
> -----------------------------------------
>
>                 Key: SOLR-12709
>                 URL: https://issues.apache.org/jira/browse/SOLR-12709
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Andrzej Bialecki 
>            Assignee: Andrzej Bialecki 
>            Priority: Major
>             Fix For: 7.5, master (8.0)
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message