spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: Spark Streaming backpressure weird behavior/bug
Date Thu, 03 Nov 2016 21:45:33 GMT
Yeah, that looks pretty bad.  Have you tried just setting max rate per
partition without turning backpressure on?

If you want to keep digging on this, can you add some debugging output
related to the backpressure?

if you add a line like this to your log4j.properties

log4j.logger.org.apache.spark.streaming.scheduler.rate=TRACE

you should start seeing log lines like

16/10/12 12:18:01 TRACE PIDRateEstimator:
time = 1476292681092, # records = 20, processing time = 20949,
scheduling delay = 6
16/10/12 12:18:01 TRACE PIDRateEstimator:
latestRate = -1.0, error = -1.9546995083297531
latestError = -1.0, historicalError = 0.001145639409995704
delaySinceUpdate = 1.476292681093E9, dError = -6.466871512381435E-10

and then once it updates, lines like

16/10/12 12:18:32 TRACE PIDRateEstimator: New rate = 1.0

On Wed, Nov 2, 2016 at 9:43 PM, map reduced <k3t.git.1@gmail.com> wrote:

> It happened again (this time i've got the partitions too from the logs) -
> 2 billion batch size all of a sudden!
>
> [image: Inline image 1]
>
>
> topic: kafka_topic_A    partition: 51    offsets: 1020742738 to 1029289633
> topic: kafka_topic_A    partition: 101    offsets: 1020736302 to 1029287024
> topic: kafka_topic_A    partition: 58    offsets: 1020777070 to 1029332079
> topic: kafka_topic_B    partition: 4    offsets: 4803171900 to 4813684863
> topic: kafka_topic_A    partition: 181    offsets: 1020695323 to 1029247077
> topic: kafka_topic_A    partition: 120    offsets: 1020843047 to 1029392933
> topic: kafka_topic_A    partition: 21    offsets: 24723134979 to
> 24731684016
> topic: kafka_topic_A    partition: 232    offsets: 1020850783 to 1029399540
> topic: kafka_topic_A    partition: 140    offsets: 1020857031 to 1029409063
> topic: kafka_topic_A    partition: 24    offsets: 24727354514 to
> 24735900600
> topic: kafka_topic_A    partition: 27    offsets: 24707635520 to
> 24716178579
> topic: kafka_topic_A    partition: 108    offsets: 1020522661 to 1029068390
> topic: kafka_topic_A    partition: 67    offsets: 1020836326 to 1029387310
> topic: kafka_topic_A    partition: 243    offsets: 1020719277 to 1029269108
> topic: kafka_topic_A    partition: 222    offsets: 1020842498 to 1029394654
> topic: kafka_topic_A    partition: 42    offsets: 24717681095 to
> 24726227066
> topic: kafka_topic_A    partition: 23    offsets: 24729438206 to
> 24737988239
> topic: kafka_topic_A    partition: 119    offsets: 1020720387 to 1029268682
> topic: kafka_topic_B    partition: 37    offsets: 4801248272 to 4811770427
> topic: kafka_topic_B    partition: 38    offsets: 4802833315 to 4813345630
> topic: kafka_topic_A    partition: 244    offsets: 1021008217 to 1029563278
> topic: kafka_topic_A    partition: 203    offsets: 1020670345 to 1029221218
> topic: kafka_topic_A    partition: 66    offsets: 1020747290 to 1029293991
> topic: kafka_topic_A    partition: 165    offsets: 1020857985 to 1029408487
> topic: kafka_topic_A    partition: 110    offsets: 1020791425 to 1029339894
> topic: kafka_topic_A    partition: 150    offsets: 1020714886 to 1029263887
> topic: kafka_topic_A    partition: 85    offsets: 1020667473 to 1029213323
> topic: kafka_topic_A    partition: 105    offsets: 1020939489 to 1029488428
> topic: kafka_topic_A    partition: 72    offsets: 1020837820 to 1029389538
> topic: kafka_topic_A    partition: 146    offsets: 1020770790 to 1029320327
> topic: kafka_topic_A    partition: 90    offsets: 1020826980 to 1029375310
> topic: kafka_topic_A    partition: 138    offsets: 1020813165 to 1029364755
> topic: kafka_topic_B    partition: 18    offsets: 4801290926 to 4811805578
> topic: kafka_topic_B    partition: 1    offsets: 4802397679 to 4812912703
> topic: kafka_topic_A    partition: 182    offsets: 1020944719 to 1029494237
> topic: kafka_topic_B    partition: 5    offsets: 4808767497 to 4819286328
> topic: kafka_topic_A    partition: 199    offsets: 1020828483 to 1029379310
> topic: kafka_topic_B    partition: 19    offsets: 4814797257 to 4825312689
> topic: kafka_topic_B    partition: 7    offsets: 4804013760 to 4814536974
> topic: kafka_topic_B    partition: 42    offsets: 4803850389 to 4814365291
> topic: kafka_topic_A    partition: 235    offsets: 1020692000 to 1029240754
> topic: kafka_topic_A    partition: 195    offsets: 1020779755 to 1029331674
> topic: kafka_topic_A    partition: 248    offsets: 1020644404 to 1029194743
> topic: kafka_topic_B    partition: 27    offsets: 4803952312 to 4814465967
> topic: kafka_topic_A    partition: 136    offsets: 1020801813 to 1029356188
> topic: kafka_topic_B    partition: 16    offsets: 4800603225 to 4811123659
> topic: kafka_topic_A    partition: 48    offsets: 24733300757 to
> 24741850194
> topic: kafka_topic_A    partition: 172    offsets: 1020775005 to 1029324739
> topic: kafka_topic_B    partition: 49    offsets: 4800717219 to 4811236254
> topic: kafka_topic_A    partition: 93    offsets: 1020985565 to 1029537168
> topic: kafka_topic_B    partition: 24    offsets: 4799098477 to 4809607456
> topic: kafka_topic_A    partition: 154    offsets: 1020693541 to 1029238078
> topic: kafka_topic_A    partition: 233    offsets: 1020946888 to 1029497894
> topic: kafka_topic_A    partition: 189    offsets: 1020961477 to 1029514103
> topic: kafka_topic_A    partition: 1    offsets: 24740548920 to 24749096350
> topic: kafka_topic_A    partition: 38    offsets: 24723357288 to
> 24731912319
> topic: kafka_topic_A    partition: 22    offsets: 24724263711 to
> 24732813058
> topic: kafka_topic_A    partition: 40    offsets: 24731873161 to
> 24740422207
> topic: kafka_topic_A    partition: 116    offsets: 1020576557 to 1029122423
> topic: kafka_topic_B    partition: 8    offsets: 4799369592 to 4809890388
> topic: kafka_topic_A    partition: 36    offsets: 24726594785 to
> 24735140031
> topic: kafka_topic_A    partition: 211    offsets: 1020900478 to 1029446732
> topic: kafka_topic_A    partition: 153    offsets: 1020751649 to 1029305015
> topic: kafka_topic_A    partition: 168    offsets: 1020768581 to 1029315536
> topic: kafka_topic_A    partition: 117    offsets: 1020620278 to 1029167248
> topic: kafka_topic_B    partition: 35    offsets: 4806178047 to 4816695731
> topic: kafka_topic_A    partition: 220    offsets: 1020814844 to 1029362554
> topic: kafka_topic_A    partition: 196    offsets: 1020651090 to 1029194969
> topic: kafka_topic_A    partition: 236    offsets: 1020692222 to 1029241847
> topic: kafka_topic_A    partition: 6    offsets: 24722380773 to 24730930570
> topic: kafka_topic_A    partition: 59    offsets: 1020835730 to 1029384973
> topic: kafka_topic_A    partition: 30    offsets: 24726641150 to
> 24735187702
> topic: kafka_topic_A    partition: 209    offsets: 1020874558 to 1029427895
> topic: kafka_topic_A    partition: 163    offsets: 1020703633 to 1029253408
> topic: kafka_topic_B    partition: 47    offsets: 4800171361 to 4810686521
> topic: kafka_topic_A    partition: 97    offsets: 1020667468 to 1029213541
> topic: kafka_topic_A    partition: 226    offsets: 1020960455 to 1029512858
> topic: kafka_topic_A    partition: 208    offsets: 1020884227 to 1029435364
> topic: kafka_topic_A    partition: 194    offsets: 1020964717 to 1029518958
> topic: kafka_topic_A    partition: 178    offsets: 1020632536 to 1029178618
> topic: kafka_topic_A    partition: 52    offsets: 1020842987 to 1029393669
> topic: kafka_topic_A    partition: 5    offsets: 24719725869 to 24728274543
> topic: kafka_topic_A    partition: 63    offsets: 1020887251 to 1029437144
> topic: kafka_topic_B    partition: 36    offsets: 4800982281 to 4811501000
> topic: kafka_topic_A    partition: 11    offsets: 24729694196 to
> 24738244559
> topic: kafka_topic_A    partition: 69    offsets: 1020732826 to 1029275514
> topic: kafka_topic_A    partition: 89    offsets: 1020642269 to 1029187888
> topic: kafka_topic_B    partition: 11    offsets: 4808218495 to 4818733612
> topic: kafka_topic_B    partition: 25    offsets: 4798933350 to 4809448450
> topic: kafka_topic_A    partition: 96    offsets: 1020846117 to 1029393750
> topic: kafka_topic_B    partition: 10    offsets: 4803818779 to 4814337498
> topic: kafka_topic_A    partition: 37    offsets: 24739837165 to
> 24748391468
> topic: kafka_topic_B    partition: 32    offsets: 4810693793 to 4821217501
> topic: kafka_topic_A    partition: 134    offsets: 1020747722 to 1029296407
> topic: kafka_topic_A    partition: 13    offsets: 24734355357 to
> 24742905825
> topic: kafka_topic_A    partition: 19    offsets: 24732775735 to
> 24741322331
> topic: kafka_topic_A    partition: 229    offsets: 1020798266 to 1029347927
> topic: kafka_topic_A    partition: 91    offsets: 1020974276 to 1029525120
> topic: kafka_topic_A    partition: 64    offsets: 1020980318 to 1029530189
> topic: kafka_topic_A    partition: 34    offsets: 24723495628 to
> 24732054835
> topic: kafka_topic_A    partition: 4    offsets: 24727632125 to 24736184191
> topic: kafka_topic_A    partition: 175    offsets: 1020915534 to 1029464464
> topic: kafka_topic_A    partition: 53    offsets: 1020704573 to 1029254608
> topic: kafka_topic_A    partition: 143    offsets: 1020772985 to 1029322428
> topic: kafka_topic_A    partition: 118    offsets: 1020778666 to 1029331391
> topic: kafka_topic_A    partition: 249    offsets: 1020963635 to 1029516291
> topic: kafka_topic_A    partition: 3    offsets: 24721520599 to 24730075720
> topic: kafka_topic_A    partition: 184    offsets: 1020775444 to 1029326031
> topic: kafka_topic_A    partition: 225    offsets: 1020933583 to 1029483635
> topic: kafka_topic_A    partition: 188    offsets: 1020647943 to 1029198446
> topic: kafka_topic_A    partition: 94    offsets: 1020730941 to 1029278716
> topic: kafka_topic_A    partition: 213    offsets: 1020762226 to 1029311435
> topic: kafka_topic_A    partition: 151    offsets: 1020844374 to 1029395379
> topic: kafka_topic_A    partition: 125    offsets: 1020760525 to 1029306817
> topic: kafka_topic_A    partition: 139    offsets: 1020830596 to 1029382287
> topic: kafka_topic_A    partition: 223    offsets: 1020851931 to 1029406373
> topic: kafka_topic_A    partition: 79    offsets: 1020569596 to 1029117673
> topic: kafka_topic_B    partition: 41    offsets: 4802503055 to 4813020137
> topic: kafka_topic_A    partition: 157    offsets: 1020773259 to 1029323214
> topic: kafka_topic_B    partition: 43    offsets: 4807530119 to 4818051823
> topic: kafka_topic_B    partition: 9    offsets: 4801124375 to 4811641360
> topic: kafka_topic_A    partition: 121    offsets: 1020716814 to 1029262616
> topic: kafka_topic_A    partition: 78    offsets: 1020757202 to 1029307937
> topic: kafka_topic_A    partition: 43    offsets: 24728638290 to
> 24737193015
> topic: kafka_topic_A    partition: 113    offsets: 1020840637 to 1029386523
> topic: kafka_topic_A    partition: 219    offsets: 1020867425 to 1029414624
> topic: kafka_topic_A    partition: 17    offsets: 24719427351 to
> 24727972412
> topic: kafka_topic_A    partition: 156    offsets: 1020795237 to 1029341015
> topic: kafka_topic_A    partition: 70    offsets: 1020706495 to 1029254472
> topic: kafka_topic_A    partition: 61    offsets: 1021026951 to 1029582817
> topic: kafka_topic_A    partition: 190    offsets: 1020963590 to 1029516326
> topic: kafka_topic_A    partition: 29    offsets: 24722142896 to
> 24730694155
> topic: kafka_topic_A    partition: 207    offsets: 1020639874 to 1029187494
> topic: kafka_topic_A    partition: 177    offsets: 1020685282 to 1029233121
> topic: kafka_topic_A    partition: 160    offsets: 1020789969 to 1029337510
> topic: kafka_topic_A    partition: 102    offsets: 1020963819 to 1029516283
> topic: kafka_topic_B    partition: 20    offsets: 4801028715 to 4811550727
> topic: kafka_topic_B    partition: 13    offsets: 4797383641 to 4807902682
> topic: kafka_topic_A    partition: 128    offsets: 1020662803 to 1029211499
> topic: kafka_topic_A    partition: 215    offsets: 1020837321 to 1029389104
> topic: kafka_topic_A    partition: 240    offsets: 1021021049 to 1029572788
> topic: kafka_topic_A    partition: 56    offsets: 1020941937 to 1029496916
> topic: kafka_topic_A    partition: 147    offsets: 1020755896 to 1029303241
> topic: kafka_topic_A    partition: 112    offsets: 1020892430 to 1029441614
> topic: kafka_topic_A    partition: 45    offsets: 24716641715 to
> 24725192614
> topic: kafka_topic_A    partition: 68    offsets: 1020893444 to 1029446558
> topic: kafka_topic_A    partition: 77    offsets: 1020868499 to 1029417133
> topic: kafka_topic_B    partition: 28    offsets: 4805914153 to 4816430998
> topic: kafka_topic_A    partition: 161    offsets: 1020902852 to 1029456951
> topic: kafka_topic_A    partition: 186    offsets: 1020775276 to 1029328133
> topic: kafka_topic_B    partition: 14    offsets: 4796300859 to 4806817229
> topic: kafka_topic_A    partition: 44    offsets: 24731321741 to
> 24739866858
> topic: kafka_topic_A    partition: 47    offsets: 24726144390 to
> 24734696944
> topic: kafka_topic_A    partition: 86    offsets: 1020778038 to 1029327512
> topic: kafka_topic_A    partition: 46    offsets: 24721377928 to
> 24729930715
> topic: kafka_topic_A    partition: 200    offsets: 1020776353 to 1029328471
> topic: kafka_topic_A    partition: 132    offsets: 1020794282 to 1029343725
> topic: kafka_topic_A    partition: 100    offsets: 1020931503 to 1029480173
> topic: kafka_topic_A    partition: 212    offsets: 1020752903 to 1029303842
> topic: kafka_topic_A    partition: 193    offsets: 1020799750 to 1029348032
> topic: kafka_topic_A    partition: 239    offsets: 1020740938 to 1029296021
> topic: kafka_topic_A    partition: 242    offsets: 1021023598 to 1029575545
> topic: kafka_topic_B    partition: 40    offsets: 4801026818 to 4811537565
> topic: kafka_topic_B    partition: 12    offsets: 4798606447 to 4809123173
> topic: kafka_topic_A    partition: 18    offsets: 24725102864 to
> 24733647562
> topic: kafka_topic_A    partition: 33    offsets: 24729427865 to
> 24737975446
> topic: kafka_topic_A    partition: 16    offsets: 24725461165 to
> 24734010070
> topic: kafka_topic_A    partition: 234    offsets: 1020679052 to 1029226903
> topic: kafka_topic_A    partition: 127    offsets: 1020876420 to 1029425258
> topic: kafka_topic_A    partition: 173    offsets: 1020875774 to 1029427802
> topic: kafka_topic_A    partition: 174    offsets: 1020764367 to 1029311197
> topic: kafka_topic_A    partition: 60    offsets: 1020729422 to 1029280479
> topic: kafka_topic_A    partition: 164    offsets: 1020895388 to 1029447072
> topic: kafka_topic_B    partition: 3    offsets: 4801150811 to 4811667621
> topic: kafka_topic_A    partition: 76    offsets: 1020872633 to 1029425200
> topic: kafka_topic_A    partition: 2    offsets: 24720552836 to 24729103435
> topic: kafka_topic_A    partition: 31    offsets: 24724971328 to
> 24733525699
> topic: kafka_topic_A    partition: 180    offsets: 1020790913 to 1029342607
> topic: kafka_topic_A    partition: 7    offsets: 24722917305 to 24731461090
> topic: kafka_topic_A    partition: 0    offsets: 24715978894 to 24724533838
> topic: kafka_topic_B    partition: 6    offsets: 4801685031 to 4812197203
> topic: kafka_topic_A    partition: 111    offsets: 1020777248 to 1029320002
> topic: kafka_topic_A    partition: 214    offsets: 1020847267 to 1029397260
> topic: kafka_topic_A    partition: 183    offsets: 1020829424 to 1029374366
> topic: kafka_topic_A    partition: 247    offsets: 1020951407 to 1029501748
> topic: kafka_topic_A    partition: 35    offsets: 24724710806 to
> 24733257282
> topic: kafka_topic_B    partition: 2    offsets: 4799162386 to 4809677022
> topic: kafka_topic_B    partition: 23    offsets: 4806523148 to 4817037826
> topic: kafka_topic_A    partition: 84    offsets: 1021016106 to 1029568619
> topic: kafka_topic_B    partition: 31    offsets: 4807475059 to 4817992907
> topic: kafka_topic_A    partition: 15    offsets: 24722975566 to
> 24731525636
> topic: kafka_topic_A    partition: 238    offsets: 1020838617 to 1029388674
> topic: kafka_topic_A    partition: 217    offsets: 1020963813 to 1029516908
> topic: kafka_topic_A    partition: 141    offsets: 1020928927 to 1029480391
> topic: kafka_topic_B    partition: 21    offsets: 4799274035 to 4809790430
> topic: kafka_topic_A    partition: 142    offsets: 1020859803 to 1029410671
> topic: kafka_topic_A    partition: 26    offsets: 24716858647 to
> 24725403869
> topic: kafka_topic_A    partition: 75    offsets: 1020875615 to 1029425108
> topic: kafka_topic_A    partition: 88    offsets: 1020636598 to 1029181677
> topic: kafka_topic_A    partition: 55    offsets: 1020981245 to 1029532042
> topic: kafka_topic_B    partition: 26    offsets: 4802386319 to 4812903171
> topic: kafka_topic_A    partition: 176    offsets: 1020927564 to 1029478273
> topic: kafka_topic_A    partition: 246    offsets: 1020902960 to 1029456226
> topic: kafka_topic_A    partition: 237    offsets: 1020879351 to 1029428560
> topic: kafka_topic_A    partition: 124    offsets: 1020844750 to 1029398619
> topic: kafka_topic_A    partition: 216    offsets: 1020606507 to 1029155109
> topic: kafka_topic_A    partition: 32    offsets: 24727599739 to
> 24736149128
> topic: kafka_topic_A    partition: 25    offsets: 24740711757 to
> 24749263320
> topic: kafka_topic_A    partition: 197    offsets: 1021032158 to 1029587829
> topic: kafka_topic_B    partition: 44    offsets: 4810511791 to 4821029704
> topic: kafka_topic_A    partition: 95    offsets: 1020733833 to 1029283829
> topic: kafka_topic_A    partition: 12    offsets: 24723998129 to
> 24732553534
> topic: kafka_topic_A    partition: 109    offsets: 1020895980 to 1029446212
> topic: kafka_topic_B    partition: 22    offsets: 4801811942 to 4812330157
> topic: kafka_topic_A    partition: 135    offsets: 1020523998 to 1029067367
> topic: kafka_topic_B    partition: 48    offsets: 4805322090 to 4815838865
> topic: kafka_topic_A    partition: 74    offsets: 1020819147 to 1029369936
> topic: kafka_topic_A    partition: 230    offsets: 1020784136 to 1029333313
> topic: kafka_topic_A    partition: 103    offsets: 1020921485 to 1029473542
> topic: kafka_topic_B    partition: 34    offsets: 4801025503 to 4811545042
> topic: kafka_topic_A    partition: 115    offsets: 1020600722 to 1029148541
> topic: kafka_topic_A    partition: 152    offsets: 1020677041 to 1029226178
> topic: kafka_topic_A    partition: 158    offsets: 1020735842 to 1029285162
> topic: kafka_topic_A    partition: 210    offsets: 1020838912 to 1029389328
> topic: kafka_topic_A    partition: 123    offsets: 1020888750 to 1029442669
> topic: kafka_topic_A    partition: 49    offsets: 24733516034 to
> 24742064144
> topic: kafka_topic_B    partition: 39    offsets: 4806601961 to 4817119869
> topic: kafka_topic_A    partition: 114    offsets: 1020945219 to 1029496002
> topic: kafka_topic_A    partition: 65    offsets: 1020714711 to 1029267579
> topic: kafka_topic_A    partition: 98    offsets: 1020581086 to 1029126420
> topic: kafka_topic_B    partition: 33    offsets: 4802443872 to 4812950776
> topic: kafka_topic_A    partition: 73    offsets: 1020908814 to 1029459329
> topic: kafka_topic_A    partition: 14    offsets: 24720549899 to
> 24729100604
> topic: kafka_topic_A    partition: 106    offsets: 1020832194 to 1029381879
> topic: kafka_topic_B    partition: 46    offsets: 4805759222 to 4816272314
> topic: kafka_topic_A    partition: 130    offsets: 1020729244 to 1029276701
> topic: kafka_topic_A    partition: 166    offsets: 1020939071 to 1029489456
> topic: kafka_topic_A    partition: 104    offsets: 1020771720 to 1029318470
> topic: kafka_topic_A    partition: 224    offsets: 1021062976 to 1029618193
> topic: kafka_topic_B    partition: 0    offsets: 4805841603 to 4816356537
> topic: kafka_topic_A    partition: 39    offsets: 24733836602 to
> 24742385677
> topic: kafka_topic_A    partition: 202    offsets: 1020738496 to 1029289191
> topic: kafka_topic_A    partition: 62    offsets: 1020767369 to 1029310260
> topic: kafka_topic_A    partition: 54    offsets: 1020872832 to 1029424418
> topic: kafka_topic_A    partition: 155    offsets: 1020939790 to 1029491266
> topic: kafka_topic_A    partition: 57    offsets: 1020926473 to 1029478170
> topic: kafka_topic_A    partition: 10    offsets: 24722360402 to
> 24730916736
> topic: kafka_topic_A    partition: 227    offsets: 1020628274 to 1029175330
> topic: kafka_topic_A    partition: 205    offsets: 1020886863 to 1029438420
> topic: kafka_topic_A    partition: 9    offsets: 24730599499 to 24739147248
> topic: kafka_topic_A    partition: 218    offsets: 1020694139 to 1029244205
> topic: kafka_topic_A    partition: 81    offsets: 1020865158 to 1029417909
> topic: kafka_topic_A    partition: 99    offsets: 1020829095 to 1029378716
> topic: kafka_topic_A    partition: 144    offsets: 1020836880 to 1029390098
> topic: kafka_topic_A    partition: 80    offsets: 1020632760 to 1029181116
> topic: kafka_topic_A    partition: 185    offsets: 1020777167 to 1029326135
> topic: kafka_topic_A    partition: 137    offsets: 1020783286 to 1029336240
> topic: kafka_topic_A    partition: 145    offsets: 1020807427 to 1029353122
> topic: kafka_topic_A    partition: 122    offsets: 1020914744 to 1029465920
> topic: kafka_topic_A    partition: 133    offsets: 1020818950 to 1029369827
> topic: kafka_topic_A    partition: 71    offsets: 1020604295 to 1029151699
> topic: kafka_topic_A    partition: 82    offsets: 1020925125 to 1029478280
> topic: kafka_topic_A    partition: 87    offsets: 1020857237 to 1029406722
> topic: kafka_topic_A    partition: 201    offsets: 1020709307 to 1029260228
> topic: kafka_topic_A    partition: 28    offsets: 24728200955 to
> 24736749015
> topic: kafka_topic_A    partition: 41    offsets: 24729533353 to
> 24738085917
> topic: kafka_topic_A    partition: 170    offsets: 1020668802 to 1029219950
> topic: kafka_topic_A    partition: 187    offsets: 1020581810 to 1029129601
> topic: kafka_topic_B    partition: 29    offsets: 4803280139 to 4813797539
> topic: kafka_topic_A    partition: 92    offsets: 1020662671 to 1029214523
> topic: kafka_topic_A    partition: 231    offsets: 1020772888 to 1029320782
> topic: kafka_topic_A    partition: 241    offsets: 1020649136 to 1029195109
> topic: kafka_topic_A    partition: 192    offsets: 1020839092 to 1029389989
> topic: kafka_topic_A    partition: 8    offsets: 24732792451 to 24741339710
> topic: kafka_topic_A    partition: 131    offsets: 1020886007 to 1029433501
> topic: kafka_topic_A    partition: 162    offsets: 1020706400 to 1029251727
> topic: kafka_topic_A    partition: 126    offsets: 1020828002 to 1029377579
> topic: kafka_topic_A    partition: 228    offsets: 1020824139 to 1029371645
> topic: kafka_topic_A    partition: 167    offsets: 1020746310 to 1029296452
> topic: kafka_topic_B    partition: 30    offsets: 4795764234 to 4806277616
> topic: kafka_topic_A    partition: 221    offsets: 1020618597 to 1029166130
> topic: kafka_topic_A    partition: 206    offsets: 1020972294 to 1029522361
> topic: kafka_topic_A    partition: 245    offsets: 1020859155 to 1029409690
> topic: kafka_topic_A    partition: 148    offsets: 1020689094 to 1029234764
> topic: kafka_topic_A    partition: 171    offsets: 1020893286 to 1029448085
> topic: kafka_topic_A    partition: 20    offsets: 24727739340 to
> 24736287861
> topic: kafka_topic_A    partition: 159    offsets: 1020770845 to 1029316911
> topic: kafka_topic_A    partition: 169    offsets: 1020699633 to 1029253155
> topic: kafka_topic_A    partition: 83    offsets: 1020954835 to 1029507004
> topic: kafka_topic_A    partition: 149    offsets: 1020763182 to 1029312029
> topic: kafka_topic_B    partition: 17    offsets: 4798809279 to 4809328520
> topic: kafka_topic_A    partition: 191    offsets: 1020939618 to 1029492433
> topic: kafka_topic_A    partition: 50    offsets: 1020781205 to 1029327065
> topic: kafka_topic_A    partition: 107    offsets: 1020596042 to 1029143966
> topic: kafka_topic_A    partition: 179    offsets: 1020692875 to 1029239892
> topic: kafka_topic_A    partition: 204    offsets: 1020682012 to 1029229892
> topic: kafka_topic_B    partition: 15    offsets: 4797528038 to 4808038327
> topic: kafka_topic_A    partition: 198    offsets: 1020530213 to 1029075405
> topic: kafka_topic_B    partition: 45    offsets: 4803051802 to 4813564524
> topic: kafka_topic_A    partition: 129    offsets: 1020804825 to 1029355767
>
>
> On Wed, Nov 2, 2016 at 11:21 AM, map reduced <k3t.git.1@gmail.com> wrote:
>
>> Yes it does, I checked in the logs. Infact, if you see the first
>> screenshot, stream processing was 'stuck' processing those many records for
>> quite some time (~ 1hr).
>> One thing I noticed is initial batches took (maybe far?) longer than the
>> configured batchDuration of 1.5mins, say in case screenshot 2, it took
>> 5.8-7.1min and in case 1 it took 3-4 mins.
>>
>> On Wed, Nov 2, 2016 at 8:43 AM, Cody Koeninger <cody@koeninger.org>
>> wrote:
>>
>>> Does that batch actually have that many records in it (you should be
>>> able to see beginning and ending offsets in the logs), or is it an error in
>>> the UI?
>>>
>>>
>>> On Tue, Nov 1, 2016 at 11:59 PM, map reduced <k3t.git.1@gmail.com>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I am using Spark 2.0.0 standalone cluster, regular streaming job
>>>> consuming from kafka and writing to http endpoint. I have configuration:
>>>> executors 7 cores/executor, maxCores = 84 (so 12 executors)
>>>> batchsize - 90 seconds
>>>> maxRatePerPartition - 2000
>>>> backPressure enabled = true
>>>>
>>>> My kafka topics have total of 300 partitions, so I am expecting to be
>>>> max 54million records per batch (maxRatePerPartition * batchsize *
>>>> #partitions) - and that's what I am getting. But it turns out that it can't
>>>> process 54million records in 90sec batch, so I am expecting backpressure
to
>>>> kick in, but I see something strange there. It reduces batch size to lesser
>>>> # of records, but then suddenly spits out a HUGE batch size of 13 billion
>>>> records.
>>>>
>>>> [image: Inline image 1]
>>>> I changed some configuration to see if above was a one off case but the
>>>> same issue happened again. Check the below screenshot (huge batch size of
>>>> 14 billion records again!) :
>>>>
>>>> [image: Inline image 2]
>>>>
>>>> Is this a bug? Any reasoning you know for this to happen?
>>>>
>>>> Thanks,
>>>> KP
>>>>
>>>
>>>
>>
>

Mime
View raw message