spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From map reduced <k3t.gi...@gmail.com>
Subject Re: Spark Streaming backpressure weird behavior/bug
Date Thu, 03 Nov 2016 02:43:16 GMT
It happened again (this time i've got the partitions too from the logs) - 2
billion batch size all of a sudden!

[image: Inline image 1]


topic: kafka_topic_A    partition: 51    offsets: 1020742738 to 1029289633
topic: kafka_topic_A    partition: 101    offsets: 1020736302 to 1029287024
topic: kafka_topic_A    partition: 58    offsets: 1020777070 to 1029332079
topic: kafka_topic_B    partition: 4    offsets: 4803171900 to 4813684863
topic: kafka_topic_A    partition: 181    offsets: 1020695323 to 1029247077
topic: kafka_topic_A    partition: 120    offsets: 1020843047 to 1029392933
topic: kafka_topic_A    partition: 21    offsets: 24723134979 to 24731684016
topic: kafka_topic_A    partition: 232    offsets: 1020850783 to 1029399540
topic: kafka_topic_A    partition: 140    offsets: 1020857031 to 1029409063
topic: kafka_topic_A    partition: 24    offsets: 24727354514 to 24735900600
topic: kafka_topic_A    partition: 27    offsets: 24707635520 to 24716178579
topic: kafka_topic_A    partition: 108    offsets: 1020522661 to 1029068390
topic: kafka_topic_A    partition: 67    offsets: 1020836326 to 1029387310
topic: kafka_topic_A    partition: 243    offsets: 1020719277 to 1029269108
topic: kafka_topic_A    partition: 222    offsets: 1020842498 to 1029394654
topic: kafka_topic_A    partition: 42    offsets: 24717681095 to 24726227066
topic: kafka_topic_A    partition: 23    offsets: 24729438206 to 24737988239
topic: kafka_topic_A    partition: 119    offsets: 1020720387 to 1029268682
topic: kafka_topic_B    partition: 37    offsets: 4801248272 to 4811770427
topic: kafka_topic_B    partition: 38    offsets: 4802833315 to 4813345630
topic: kafka_topic_A    partition: 244    offsets: 1021008217 to 1029563278
topic: kafka_topic_A    partition: 203    offsets: 1020670345 to 1029221218
topic: kafka_topic_A    partition: 66    offsets: 1020747290 to 1029293991
topic: kafka_topic_A    partition: 165    offsets: 1020857985 to 1029408487
topic: kafka_topic_A    partition: 110    offsets: 1020791425 to 1029339894
topic: kafka_topic_A    partition: 150    offsets: 1020714886 to 1029263887
topic: kafka_topic_A    partition: 85    offsets: 1020667473 to 1029213323
topic: kafka_topic_A    partition: 105    offsets: 1020939489 to 1029488428
topic: kafka_topic_A    partition: 72    offsets: 1020837820 to 1029389538
topic: kafka_topic_A    partition: 146    offsets: 1020770790 to 1029320327
topic: kafka_topic_A    partition: 90    offsets: 1020826980 to 1029375310
topic: kafka_topic_A    partition: 138    offsets: 1020813165 to 1029364755
topic: kafka_topic_B    partition: 18    offsets: 4801290926 to 4811805578
topic: kafka_topic_B    partition: 1    offsets: 4802397679 to 4812912703
topic: kafka_topic_A    partition: 182    offsets: 1020944719 to 1029494237
topic: kafka_topic_B    partition: 5    offsets: 4808767497 to 4819286328
topic: kafka_topic_A    partition: 199    offsets: 1020828483 to 1029379310
topic: kafka_topic_B    partition: 19    offsets: 4814797257 to 4825312689
topic: kafka_topic_B    partition: 7    offsets: 4804013760 to 4814536974
topic: kafka_topic_B    partition: 42    offsets: 4803850389 to 4814365291
topic: kafka_topic_A    partition: 235    offsets: 1020692000 to 1029240754
topic: kafka_topic_A    partition: 195    offsets: 1020779755 to 1029331674
topic: kafka_topic_A    partition: 248    offsets: 1020644404 to 1029194743
topic: kafka_topic_B    partition: 27    offsets: 4803952312 to 4814465967
topic: kafka_topic_A    partition: 136    offsets: 1020801813 to 1029356188
topic: kafka_topic_B    partition: 16    offsets: 4800603225 to 4811123659
topic: kafka_topic_A    partition: 48    offsets: 24733300757 to 24741850194
topic: kafka_topic_A    partition: 172    offsets: 1020775005 to 1029324739
topic: kafka_topic_B    partition: 49    offsets: 4800717219 to 4811236254
topic: kafka_topic_A    partition: 93    offsets: 1020985565 to 1029537168
topic: kafka_topic_B    partition: 24    offsets: 4799098477 to 4809607456
topic: kafka_topic_A    partition: 154    offsets: 1020693541 to 1029238078
topic: kafka_topic_A    partition: 233    offsets: 1020946888 to 1029497894
topic: kafka_topic_A    partition: 189    offsets: 1020961477 to 1029514103
topic: kafka_topic_A    partition: 1    offsets: 24740548920 to 24749096350
topic: kafka_topic_A    partition: 38    offsets: 24723357288 to 24731912319
topic: kafka_topic_A    partition: 22    offsets: 24724263711 to 24732813058
topic: kafka_topic_A    partition: 40    offsets: 24731873161 to 24740422207
topic: kafka_topic_A    partition: 116    offsets: 1020576557 to 1029122423
topic: kafka_topic_B    partition: 8    offsets: 4799369592 to 4809890388
topic: kafka_topic_A    partition: 36    offsets: 24726594785 to 24735140031
topic: kafka_topic_A    partition: 211    offsets: 1020900478 to 1029446732
topic: kafka_topic_A    partition: 153    offsets: 1020751649 to 1029305015
topic: kafka_topic_A    partition: 168    offsets: 1020768581 to 1029315536
topic: kafka_topic_A    partition: 117    offsets: 1020620278 to 1029167248
topic: kafka_topic_B    partition: 35    offsets: 4806178047 to 4816695731
topic: kafka_topic_A    partition: 220    offsets: 1020814844 to 1029362554
topic: kafka_topic_A    partition: 196    offsets: 1020651090 to 1029194969
topic: kafka_topic_A    partition: 236    offsets: 1020692222 to 1029241847
topic: kafka_topic_A    partition: 6    offsets: 24722380773 to 24730930570
topic: kafka_topic_A    partition: 59    offsets: 1020835730 to 1029384973
topic: kafka_topic_A    partition: 30    offsets: 24726641150 to 24735187702
topic: kafka_topic_A    partition: 209    offsets: 1020874558 to 1029427895
topic: kafka_topic_A    partition: 163    offsets: 1020703633 to 1029253408
topic: kafka_topic_B    partition: 47    offsets: 4800171361 to 4810686521
topic: kafka_topic_A    partition: 97    offsets: 1020667468 to 1029213541
topic: kafka_topic_A    partition: 226    offsets: 1020960455 to 1029512858
topic: kafka_topic_A    partition: 208    offsets: 1020884227 to 1029435364
topic: kafka_topic_A    partition: 194    offsets: 1020964717 to 1029518958
topic: kafka_topic_A    partition: 178    offsets: 1020632536 to 1029178618
topic: kafka_topic_A    partition: 52    offsets: 1020842987 to 1029393669
topic: kafka_topic_A    partition: 5    offsets: 24719725869 to 24728274543
topic: kafka_topic_A    partition: 63    offsets: 1020887251 to 1029437144
topic: kafka_topic_B    partition: 36    offsets: 4800982281 to 4811501000
topic: kafka_topic_A    partition: 11    offsets: 24729694196 to 24738244559
topic: kafka_topic_A    partition: 69    offsets: 1020732826 to 1029275514
topic: kafka_topic_A    partition: 89    offsets: 1020642269 to 1029187888
topic: kafka_topic_B    partition: 11    offsets: 4808218495 to 4818733612
topic: kafka_topic_B    partition: 25    offsets: 4798933350 to 4809448450
topic: kafka_topic_A    partition: 96    offsets: 1020846117 to 1029393750
topic: kafka_topic_B    partition: 10    offsets: 4803818779 to 4814337498
topic: kafka_topic_A    partition: 37    offsets: 24739837165 to 24748391468
topic: kafka_topic_B    partition: 32    offsets: 4810693793 to 4821217501
topic: kafka_topic_A    partition: 134    offsets: 1020747722 to 1029296407
topic: kafka_topic_A    partition: 13    offsets: 24734355357 to 24742905825
topic: kafka_topic_A    partition: 19    offsets: 24732775735 to 24741322331
topic: kafka_topic_A    partition: 229    offsets: 1020798266 to 1029347927
topic: kafka_topic_A    partition: 91    offsets: 1020974276 to 1029525120
topic: kafka_topic_A    partition: 64    offsets: 1020980318 to 1029530189
topic: kafka_topic_A    partition: 34    offsets: 24723495628 to 24732054835
topic: kafka_topic_A    partition: 4    offsets: 24727632125 to 24736184191
topic: kafka_topic_A    partition: 175    offsets: 1020915534 to 1029464464
topic: kafka_topic_A    partition: 53    offsets: 1020704573 to 1029254608
topic: kafka_topic_A    partition: 143    offsets: 1020772985 to 1029322428
topic: kafka_topic_A    partition: 118    offsets: 1020778666 to 1029331391
topic: kafka_topic_A    partition: 249    offsets: 1020963635 to 1029516291
topic: kafka_topic_A    partition: 3    offsets: 24721520599 to 24730075720
topic: kafka_topic_A    partition: 184    offsets: 1020775444 to 1029326031
topic: kafka_topic_A    partition: 225    offsets: 1020933583 to 1029483635
topic: kafka_topic_A    partition: 188    offsets: 1020647943 to 1029198446
topic: kafka_topic_A    partition: 94    offsets: 1020730941 to 1029278716
topic: kafka_topic_A    partition: 213    offsets: 1020762226 to 1029311435
topic: kafka_topic_A    partition: 151    offsets: 1020844374 to 1029395379
topic: kafka_topic_A    partition: 125    offsets: 1020760525 to 1029306817
topic: kafka_topic_A    partition: 139    offsets: 1020830596 to 1029382287
topic: kafka_topic_A    partition: 223    offsets: 1020851931 to 1029406373
topic: kafka_topic_A    partition: 79    offsets: 1020569596 to 1029117673
topic: kafka_topic_B    partition: 41    offsets: 4802503055 to 4813020137
topic: kafka_topic_A    partition: 157    offsets: 1020773259 to 1029323214
topic: kafka_topic_B    partition: 43    offsets: 4807530119 to 4818051823
topic: kafka_topic_B    partition: 9    offsets: 4801124375 to 4811641360
topic: kafka_topic_A    partition: 121    offsets: 1020716814 to 1029262616
topic: kafka_topic_A    partition: 78    offsets: 1020757202 to 1029307937
topic: kafka_topic_A    partition: 43    offsets: 24728638290 to 24737193015
topic: kafka_topic_A    partition: 113    offsets: 1020840637 to 1029386523
topic: kafka_topic_A    partition: 219    offsets: 1020867425 to 1029414624
topic: kafka_topic_A    partition: 17    offsets: 24719427351 to 24727972412
topic: kafka_topic_A    partition: 156    offsets: 1020795237 to 1029341015
topic: kafka_topic_A    partition: 70    offsets: 1020706495 to 1029254472
topic: kafka_topic_A    partition: 61    offsets: 1021026951 to 1029582817
topic: kafka_topic_A    partition: 190    offsets: 1020963590 to 1029516326
topic: kafka_topic_A    partition: 29    offsets: 24722142896 to 24730694155
topic: kafka_topic_A    partition: 207    offsets: 1020639874 to 1029187494
topic: kafka_topic_A    partition: 177    offsets: 1020685282 to 1029233121
topic: kafka_topic_A    partition: 160    offsets: 1020789969 to 1029337510
topic: kafka_topic_A    partition: 102    offsets: 1020963819 to 1029516283
topic: kafka_topic_B    partition: 20    offsets: 4801028715 to 4811550727
topic: kafka_topic_B    partition: 13    offsets: 4797383641 to 4807902682
topic: kafka_topic_A    partition: 128    offsets: 1020662803 to 1029211499
topic: kafka_topic_A    partition: 215    offsets: 1020837321 to 1029389104
topic: kafka_topic_A    partition: 240    offsets: 1021021049 to 1029572788
topic: kafka_topic_A    partition: 56    offsets: 1020941937 to 1029496916
topic: kafka_topic_A    partition: 147    offsets: 1020755896 to 1029303241
topic: kafka_topic_A    partition: 112    offsets: 1020892430 to 1029441614
topic: kafka_topic_A    partition: 45    offsets: 24716641715 to 24725192614
topic: kafka_topic_A    partition: 68    offsets: 1020893444 to 1029446558
topic: kafka_topic_A    partition: 77    offsets: 1020868499 to 1029417133
topic: kafka_topic_B    partition: 28    offsets: 4805914153 to 4816430998
topic: kafka_topic_A    partition: 161    offsets: 1020902852 to 1029456951
topic: kafka_topic_A    partition: 186    offsets: 1020775276 to 1029328133
topic: kafka_topic_B    partition: 14    offsets: 4796300859 to 4806817229
topic: kafka_topic_A    partition: 44    offsets: 24731321741 to 24739866858
topic: kafka_topic_A    partition: 47    offsets: 24726144390 to 24734696944
topic: kafka_topic_A    partition: 86    offsets: 1020778038 to 1029327512
topic: kafka_topic_A    partition: 46    offsets: 24721377928 to 24729930715
topic: kafka_topic_A    partition: 200    offsets: 1020776353 to 1029328471
topic: kafka_topic_A    partition: 132    offsets: 1020794282 to 1029343725
topic: kafka_topic_A    partition: 100    offsets: 1020931503 to 1029480173
topic: kafka_topic_A    partition: 212    offsets: 1020752903 to 1029303842
topic: kafka_topic_A    partition: 193    offsets: 1020799750 to 1029348032
topic: kafka_topic_A    partition: 239    offsets: 1020740938 to 1029296021
topic: kafka_topic_A    partition: 242    offsets: 1021023598 to 1029575545
topic: kafka_topic_B    partition: 40    offsets: 4801026818 to 4811537565
topic: kafka_topic_B    partition: 12    offsets: 4798606447 to 4809123173
topic: kafka_topic_A    partition: 18    offsets: 24725102864 to 24733647562
topic: kafka_topic_A    partition: 33    offsets: 24729427865 to 24737975446
topic: kafka_topic_A    partition: 16    offsets: 24725461165 to 24734010070
topic: kafka_topic_A    partition: 234    offsets: 1020679052 to 1029226903
topic: kafka_topic_A    partition: 127    offsets: 1020876420 to 1029425258
topic: kafka_topic_A    partition: 173    offsets: 1020875774 to 1029427802
topic: kafka_topic_A    partition: 174    offsets: 1020764367 to 1029311197
topic: kafka_topic_A    partition: 60    offsets: 1020729422 to 1029280479
topic: kafka_topic_A    partition: 164    offsets: 1020895388 to 1029447072
topic: kafka_topic_B    partition: 3    offsets: 4801150811 to 4811667621
topic: kafka_topic_A    partition: 76    offsets: 1020872633 to 1029425200
topic: kafka_topic_A    partition: 2    offsets: 24720552836 to 24729103435
topic: kafka_topic_A    partition: 31    offsets: 24724971328 to 24733525699
topic: kafka_topic_A    partition: 180    offsets: 1020790913 to 1029342607
topic: kafka_topic_A    partition: 7    offsets: 24722917305 to 24731461090
topic: kafka_topic_A    partition: 0    offsets: 24715978894 to 24724533838
topic: kafka_topic_B    partition: 6    offsets: 4801685031 to 4812197203
topic: kafka_topic_A    partition: 111    offsets: 1020777248 to 1029320002
topic: kafka_topic_A    partition: 214    offsets: 1020847267 to 1029397260
topic: kafka_topic_A    partition: 183    offsets: 1020829424 to 1029374366
topic: kafka_topic_A    partition: 247    offsets: 1020951407 to 1029501748
topic: kafka_topic_A    partition: 35    offsets: 24724710806 to 24733257282
topic: kafka_topic_B    partition: 2    offsets: 4799162386 to 4809677022
topic: kafka_topic_B    partition: 23    offsets: 4806523148 to 4817037826
topic: kafka_topic_A    partition: 84    offsets: 1021016106 to 1029568619
topic: kafka_topic_B    partition: 31    offsets: 4807475059 to 4817992907
topic: kafka_topic_A    partition: 15    offsets: 24722975566 to 24731525636
topic: kafka_topic_A    partition: 238    offsets: 1020838617 to 1029388674
topic: kafka_topic_A    partition: 217    offsets: 1020963813 to 1029516908
topic: kafka_topic_A    partition: 141    offsets: 1020928927 to 1029480391
topic: kafka_topic_B    partition: 21    offsets: 4799274035 to 4809790430
topic: kafka_topic_A    partition: 142    offsets: 1020859803 to 1029410671
topic: kafka_topic_A    partition: 26    offsets: 24716858647 to 24725403869
topic: kafka_topic_A    partition: 75    offsets: 1020875615 to 1029425108
topic: kafka_topic_A    partition: 88    offsets: 1020636598 to 1029181677
topic: kafka_topic_A    partition: 55    offsets: 1020981245 to 1029532042
topic: kafka_topic_B    partition: 26    offsets: 4802386319 to 4812903171
topic: kafka_topic_A    partition: 176    offsets: 1020927564 to 1029478273
topic: kafka_topic_A    partition: 246    offsets: 1020902960 to 1029456226
topic: kafka_topic_A    partition: 237    offsets: 1020879351 to 1029428560
topic: kafka_topic_A    partition: 124    offsets: 1020844750 to 1029398619
topic: kafka_topic_A    partition: 216    offsets: 1020606507 to 1029155109
topic: kafka_topic_A    partition: 32    offsets: 24727599739 to 24736149128
topic: kafka_topic_A    partition: 25    offsets: 24740711757 to 24749263320
topic: kafka_topic_A    partition: 197    offsets: 1021032158 to 1029587829
topic: kafka_topic_B    partition: 44    offsets: 4810511791 to 4821029704
topic: kafka_topic_A    partition: 95    offsets: 1020733833 to 1029283829
topic: kafka_topic_A    partition: 12    offsets: 24723998129 to 24732553534
topic: kafka_topic_A    partition: 109    offsets: 1020895980 to 1029446212
topic: kafka_topic_B    partition: 22    offsets: 4801811942 to 4812330157
topic: kafka_topic_A    partition: 135    offsets: 1020523998 to 1029067367
topic: kafka_topic_B    partition: 48    offsets: 4805322090 to 4815838865
topic: kafka_topic_A    partition: 74    offsets: 1020819147 to 1029369936
topic: kafka_topic_A    partition: 230    offsets: 1020784136 to 1029333313
topic: kafka_topic_A    partition: 103    offsets: 1020921485 to 1029473542
topic: kafka_topic_B    partition: 34    offsets: 4801025503 to 4811545042
topic: kafka_topic_A    partition: 115    offsets: 1020600722 to 1029148541
topic: kafka_topic_A    partition: 152    offsets: 1020677041 to 1029226178
topic: kafka_topic_A    partition: 158    offsets: 1020735842 to 1029285162
topic: kafka_topic_A    partition: 210    offsets: 1020838912 to 1029389328
topic: kafka_topic_A    partition: 123    offsets: 1020888750 to 1029442669
topic: kafka_topic_A    partition: 49    offsets: 24733516034 to 24742064144
topic: kafka_topic_B    partition: 39    offsets: 4806601961 to 4817119869
topic: kafka_topic_A    partition: 114    offsets: 1020945219 to 1029496002
topic: kafka_topic_A    partition: 65    offsets: 1020714711 to 1029267579
topic: kafka_topic_A    partition: 98    offsets: 1020581086 to 1029126420
topic: kafka_topic_B    partition: 33    offsets: 4802443872 to 4812950776
topic: kafka_topic_A    partition: 73    offsets: 1020908814 to 1029459329
topic: kafka_topic_A    partition: 14    offsets: 24720549899 to 24729100604
topic: kafka_topic_A    partition: 106    offsets: 1020832194 to 1029381879
topic: kafka_topic_B    partition: 46    offsets: 4805759222 to 4816272314
topic: kafka_topic_A    partition: 130    offsets: 1020729244 to 1029276701
topic: kafka_topic_A    partition: 166    offsets: 1020939071 to 1029489456
topic: kafka_topic_A    partition: 104    offsets: 1020771720 to 1029318470
topic: kafka_topic_A    partition: 224    offsets: 1021062976 to 1029618193
topic: kafka_topic_B    partition: 0    offsets: 4805841603 to 4816356537
topic: kafka_topic_A    partition: 39    offsets: 24733836602 to 24742385677
topic: kafka_topic_A    partition: 202    offsets: 1020738496 to 1029289191
topic: kafka_topic_A    partition: 62    offsets: 1020767369 to 1029310260
topic: kafka_topic_A    partition: 54    offsets: 1020872832 to 1029424418
topic: kafka_topic_A    partition: 155    offsets: 1020939790 to 1029491266
topic: kafka_topic_A    partition: 57    offsets: 1020926473 to 1029478170
topic: kafka_topic_A    partition: 10    offsets: 24722360402 to 24730916736
topic: kafka_topic_A    partition: 227    offsets: 1020628274 to 1029175330
topic: kafka_topic_A    partition: 205    offsets: 1020886863 to 1029438420
topic: kafka_topic_A    partition: 9    offsets: 24730599499 to 24739147248
topic: kafka_topic_A    partition: 218    offsets: 1020694139 to 1029244205
topic: kafka_topic_A    partition: 81    offsets: 1020865158 to 1029417909
topic: kafka_topic_A    partition: 99    offsets: 1020829095 to 1029378716
topic: kafka_topic_A    partition: 144    offsets: 1020836880 to 1029390098
topic: kafka_topic_A    partition: 80    offsets: 1020632760 to 1029181116
topic: kafka_topic_A    partition: 185    offsets: 1020777167 to 1029326135
topic: kafka_topic_A    partition: 137    offsets: 1020783286 to 1029336240
topic: kafka_topic_A    partition: 145    offsets: 1020807427 to 1029353122
topic: kafka_topic_A    partition: 122    offsets: 1020914744 to 1029465920
topic: kafka_topic_A    partition: 133    offsets: 1020818950 to 1029369827
topic: kafka_topic_A    partition: 71    offsets: 1020604295 to 1029151699
topic: kafka_topic_A    partition: 82    offsets: 1020925125 to 1029478280
topic: kafka_topic_A    partition: 87    offsets: 1020857237 to 1029406722
topic: kafka_topic_A    partition: 201    offsets: 1020709307 to 1029260228
topic: kafka_topic_A    partition: 28    offsets: 24728200955 to 24736749015
topic: kafka_topic_A    partition: 41    offsets: 24729533353 to 24738085917
topic: kafka_topic_A    partition: 170    offsets: 1020668802 to 1029219950
topic: kafka_topic_A    partition: 187    offsets: 1020581810 to 1029129601
topic: kafka_topic_B    partition: 29    offsets: 4803280139 to 4813797539
topic: kafka_topic_A    partition: 92    offsets: 1020662671 to 1029214523
topic: kafka_topic_A    partition: 231    offsets: 1020772888 to 1029320782
topic: kafka_topic_A    partition: 241    offsets: 1020649136 to 1029195109
topic: kafka_topic_A    partition: 192    offsets: 1020839092 to 1029389989
topic: kafka_topic_A    partition: 8    offsets: 24732792451 to 24741339710
topic: kafka_topic_A    partition: 131    offsets: 1020886007 to 1029433501
topic: kafka_topic_A    partition: 162    offsets: 1020706400 to 1029251727
topic: kafka_topic_A    partition: 126    offsets: 1020828002 to 1029377579
topic: kafka_topic_A    partition: 228    offsets: 1020824139 to 1029371645
topic: kafka_topic_A    partition: 167    offsets: 1020746310 to 1029296452
topic: kafka_topic_B    partition: 30    offsets: 4795764234 to 4806277616
topic: kafka_topic_A    partition: 221    offsets: 1020618597 to 1029166130
topic: kafka_topic_A    partition: 206    offsets: 1020972294 to 1029522361
topic: kafka_topic_A    partition: 245    offsets: 1020859155 to 1029409690
topic: kafka_topic_A    partition: 148    offsets: 1020689094 to 1029234764
topic: kafka_topic_A    partition: 171    offsets: 1020893286 to 1029448085
topic: kafka_topic_A    partition: 20    offsets: 24727739340 to 24736287861
topic: kafka_topic_A    partition: 159    offsets: 1020770845 to 1029316911
topic: kafka_topic_A    partition: 169    offsets: 1020699633 to 1029253155
topic: kafka_topic_A    partition: 83    offsets: 1020954835 to 1029507004
topic: kafka_topic_A    partition: 149    offsets: 1020763182 to 1029312029
topic: kafka_topic_B    partition: 17    offsets: 4798809279 to 4809328520
topic: kafka_topic_A    partition: 191    offsets: 1020939618 to 1029492433
topic: kafka_topic_A    partition: 50    offsets: 1020781205 to 1029327065
topic: kafka_topic_A    partition: 107    offsets: 1020596042 to 1029143966
topic: kafka_topic_A    partition: 179    offsets: 1020692875 to 1029239892
topic: kafka_topic_A    partition: 204    offsets: 1020682012 to 1029229892
topic: kafka_topic_B    partition: 15    offsets: 4797528038 to 4808038327
topic: kafka_topic_A    partition: 198    offsets: 1020530213 to 1029075405
topic: kafka_topic_B    partition: 45    offsets: 4803051802 to 4813564524
topic: kafka_topic_A    partition: 129    offsets: 1020804825 to 1029355767


On Wed, Nov 2, 2016 at 11:21 AM, map reduced <k3t.git.1@gmail.com> wrote:

> Yes it does, I checked in the logs. Infact, if you see the first
> screenshot, stream processing was 'stuck' processing those many records for
> quite some time (~ 1hr).
> One thing I noticed is initial batches took (maybe far?) longer than the
> configured batchDuration of 1.5mins, say in case screenshot 2, it took
> 5.8-7.1min and in case 1 it took 3-4 mins.
>
> On Wed, Nov 2, 2016 at 8:43 AM, Cody Koeninger <cody@koeninger.org> wrote:
>
>> Does that batch actually have that many records in it (you should be able
>> to see beginning and ending offsets in the logs), or is it an error in the
>> UI?
>>
>>
>> On Tue, Nov 1, 2016 at 11:59 PM, map reduced <k3t.git.1@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I am using Spark 2.0.0 standalone cluster, regular streaming job
>>> consuming from kafka and writing to http endpoint. I have configuration:
>>> executors 7 cores/executor, maxCores = 84 (so 12 executors)
>>> batchsize - 90 seconds
>>> maxRatePerPartition - 2000
>>> backPressure enabled = true
>>>
>>> My kafka topics have total of 300 partitions, so I am expecting to be
>>> max 54million records per batch (maxRatePerPartition * batchsize *
>>> #partitions) - and that's what I am getting. But it turns out that it can't
>>> process 54million records in 90sec batch, so I am expecting backpressure to
>>> kick in, but I see something strange there. It reduces batch size to lesser
>>> # of records, but then suddenly spits out a HUGE batch size of 13 billion
>>> records.
>>>
>>> [image: Inline image 1]
>>> I changed some configuration to see if above was a one off case but the
>>> same issue happened again. Check the below screenshot (huge batch size of
>>> 14 billion records again!) :
>>>
>>> [image: Inline image 2]
>>>
>>> Is this a bug? Any reasoning you know for this to happen?
>>>
>>> Thanks,
>>> KP
>>>
>>
>>
>

Mime
View raw message