spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From map reduced <k3t.gi...@gmail.com>
Subject Re: Spark Streaming backpressure weird behavior/bug
Date Fri, 04 Nov 2016 00:40:34 GMT
Forgot to add, I have turned off the backpressure (but kept
maxRatePerPartition) since the last email and it's not giving any giant
batches.

On Thu, Nov 3, 2016 at 5:11 PM, map reduced <k3t.git.1@gmail.com> wrote:

> I'll give it a try (may take some time, since this is production traffic,
> and nothing less than ERROR in prod, but will get back with the results).
> Also, it's happening pretty regularly, and very much reproducible.
>
> On Thu, Nov 3, 2016 at 2:45 PM, Cody Koeninger <cody@koeninger.org> wrote:
>
>> Yeah, that looks pretty bad.  Have you tried just setting max rate per
>> partition without turning backpressure on?
>>
>> If you want to keep digging on this, can you add some debugging output
>> related to the backpressure?
>>
>> if you add a line like this to your log4j.properties
>>
>> log4j.logger.org.apache.spark.streaming.scheduler.rate=TRACE
>>
>> you should start seeing log lines like
>>
>> 16/10/12 12:18:01 TRACE PIDRateEstimator:
>> time = 1476292681092, # records = 20, processing time = 20949,
>> scheduling delay = 6
>> 16/10/12 12:18:01 TRACE PIDRateEstimator:
>> latestRate = -1.0, error = -1.9546995083297531
>> latestError = -1.0, historicalError = 0.001145639409995704
>> delaySinceUpdate = 1.476292681093E9, dError = -6.466871512381435E-10
>>
>> and then once it updates, lines like
>>
>> 16/10/12 12:18:32 TRACE PIDRateEstimator: New rate = 1.0
>>
>> On Wed, Nov 2, 2016 at 9:43 PM, map reduced <k3t.git.1@gmail.com> wrote:
>>
>>> It happened again (this time i've got the partitions too from the logs)
>>> - 2 billion batch size all of a sudden!
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> topic: kafka_topic_A    partition: 51    offsets: 1020742738 to
>>> 1029289633
>>> topic: kafka_topic_A    partition: 101    offsets: 1020736302 to
>>> 1029287024
>>> topic: kafka_topic_A    partition: 58    offsets: 1020777070 to
>>> 1029332079
>>> topic: kafka_topic_B    partition: 4    offsets: 4803171900 to
>>> 4813684863
>>> topic: kafka_topic_A    partition: 181    offsets: 1020695323 to
>>> 1029247077
>>> topic: kafka_topic_A    partition: 120    offsets: 1020843047 to
>>> 1029392933
>>> topic: kafka_topic_A    partition: 21    offsets: 24723134979 to
>>> 24731684016
>>> topic: kafka_topic_A    partition: 232    offsets: 1020850783 to
>>> 1029399540
>>> topic: kafka_topic_A    partition: 140    offsets: 1020857031 to
>>> 1029409063
>>> topic: kafka_topic_A    partition: 24    offsets: 24727354514 to
>>> 24735900600
>>> topic: kafka_topic_A    partition: 27    offsets: 24707635520 to
>>> 24716178579
>>> topic: kafka_topic_A    partition: 108    offsets: 1020522661 to
>>> 1029068390
>>> topic: kafka_topic_A    partition: 67    offsets: 1020836326 to
>>> 1029387310
>>> topic: kafka_topic_A    partition: 243    offsets: 1020719277 to
>>> 1029269108
>>> topic: kafka_topic_A    partition: 222    offsets: 1020842498 to
>>> 1029394654
>>> topic: kafka_topic_A    partition: 42    offsets: 24717681095 to
>>> 24726227066
>>> topic: kafka_topic_A    partition: 23    offsets: 24729438206 to
>>> 24737988239
>>> topic: kafka_topic_A    partition: 119    offsets: 1020720387 to
>>> 1029268682
>>> topic: kafka_topic_B    partition: 37    offsets: 4801248272 to
>>> 4811770427
>>> topic: kafka_topic_B    partition: 38    offsets: 4802833315 to
>>> 4813345630
>>> topic: kafka_topic_A    partition: 244    offsets: 1021008217 to
>>> 1029563278
>>> topic: kafka_topic_A    partition: 203    offsets: 1020670345 to
>>> 1029221218
>>> topic: kafka_topic_A    partition: 66    offsets: 1020747290 to
>>> 1029293991
>>> topic: kafka_topic_A    partition: 165    offsets: 1020857985 to
>>> 1029408487
>>> topic: kafka_topic_A    partition: 110    offsets: 1020791425 to
>>> 1029339894
>>> topic: kafka_topic_A    partition: 150    offsets: 1020714886 to
>>> 1029263887
>>> topic: kafka_topic_A    partition: 85    offsets: 1020667473 to
>>> 1029213323
>>> topic: kafka_topic_A    partition: 105    offsets: 1020939489 to
>>> 1029488428
>>> topic: kafka_topic_A    partition: 72    offsets: 1020837820 to
>>> 1029389538
>>> topic: kafka_topic_A    partition: 146    offsets: 1020770790 to
>>> 1029320327
>>> topic: kafka_topic_A    partition: 90    offsets: 1020826980 to
>>> 1029375310
>>> topic: kafka_topic_A    partition: 138    offsets: 1020813165 to
>>> 1029364755
>>> topic: kafka_topic_B    partition: 18    offsets: 4801290926 to
>>> 4811805578
>>> topic: kafka_topic_B    partition: 1    offsets: 4802397679 to
>>> 4812912703
>>> topic: kafka_topic_A    partition: 182    offsets: 1020944719 to
>>> 1029494237
>>> topic: kafka_topic_B    partition: 5    offsets: 4808767497 to
>>> 4819286328
>>> topic: kafka_topic_A    partition: 199    offsets: 1020828483 to
>>> 1029379310
>>> topic: kafka_topic_B    partition: 19    offsets: 4814797257 to
>>> 4825312689
>>> topic: kafka_topic_B    partition: 7    offsets: 4804013760 to
>>> 4814536974
>>> topic: kafka_topic_B    partition: 42    offsets: 4803850389 to
>>> 4814365291
>>> topic: kafka_topic_A    partition: 235    offsets: 1020692000 to
>>> 1029240754
>>> topic: kafka_topic_A    partition: 195    offsets: 1020779755 to
>>> 1029331674
>>> topic: kafka_topic_A    partition: 248    offsets: 1020644404 to
>>> 1029194743
>>> topic: kafka_topic_B    partition: 27    offsets: 4803952312 to
>>> 4814465967
>>> topic: kafka_topic_A    partition: 136    offsets: 1020801813 to
>>> 1029356188
>>> topic: kafka_topic_B    partition: 16    offsets: 4800603225 to
>>> 4811123659
>>> topic: kafka_topic_A    partition: 48    offsets: 24733300757 to
>>> 24741850194
>>> topic: kafka_topic_A    partition: 172    offsets: 1020775005 to
>>> 1029324739
>>> topic: kafka_topic_B    partition: 49    offsets: 4800717219 to
>>> 4811236254
>>> topic: kafka_topic_A    partition: 93    offsets: 1020985565 to
>>> 1029537168
>>> topic: kafka_topic_B    partition: 24    offsets: 4799098477 to
>>> 4809607456
>>> topic: kafka_topic_A    partition: 154    offsets: 1020693541 to
>>> 1029238078
>>> topic: kafka_topic_A    partition: 233    offsets: 1020946888 to
>>> 1029497894
>>> topic: kafka_topic_A    partition: 189    offsets: 1020961477 to
>>> 1029514103
>>> topic: kafka_topic_A    partition: 1    offsets: 24740548920 to
>>> 24749096350
>>> topic: kafka_topic_A    partition: 38    offsets: 24723357288 to
>>> 24731912319
>>> topic: kafka_topic_A    partition: 22    offsets: 24724263711 to
>>> 24732813058
>>> topic: kafka_topic_A    partition: 40    offsets: 24731873161 to
>>> 24740422207
>>> topic: kafka_topic_A    partition: 116    offsets: 1020576557 to
>>> 1029122423
>>> topic: kafka_topic_B    partition: 8    offsets: 4799369592 to
>>> 4809890388
>>> topic: kafka_topic_A    partition: 36    offsets: 24726594785 to
>>> 24735140031
>>> topic: kafka_topic_A    partition: 211    offsets: 1020900478 to
>>> 1029446732
>>> topic: kafka_topic_A    partition: 153    offsets: 1020751649 to
>>> 1029305015
>>> topic: kafka_topic_A    partition: 168    offsets: 1020768581 to
>>> 1029315536
>>> topic: kafka_topic_A    partition: 117    offsets: 1020620278 to
>>> 1029167248
>>> topic: kafka_topic_B    partition: 35    offsets: 4806178047 to
>>> 4816695731
>>> topic: kafka_topic_A    partition: 220    offsets: 1020814844 to
>>> 1029362554
>>> topic: kafka_topic_A    partition: 196    offsets: 1020651090 to
>>> 1029194969
>>> topic: kafka_topic_A    partition: 236    offsets: 1020692222 to
>>> 1029241847
>>> topic: kafka_topic_A    partition: 6    offsets: 24722380773 to
>>> 24730930570
>>> topic: kafka_topic_A    partition: 59    offsets: 1020835730 to
>>> 1029384973
>>> topic: kafka_topic_A    partition: 30    offsets: 24726641150 to
>>> 24735187702
>>> topic: kafka_topic_A    partition: 209    offsets: 1020874558 to
>>> 1029427895
>>> topic: kafka_topic_A    partition: 163    offsets: 1020703633 to
>>> 1029253408
>>> topic: kafka_topic_B    partition: 47    offsets: 4800171361 to
>>> 4810686521
>>> topic: kafka_topic_A    partition: 97    offsets: 1020667468 to
>>> 1029213541
>>> topic: kafka_topic_A    partition: 226    offsets: 1020960455 to
>>> 1029512858
>>> topic: kafka_topic_A    partition: 208    offsets: 1020884227 to
>>> 1029435364
>>> topic: kafka_topic_A    partition: 194    offsets: 1020964717 to
>>> 1029518958
>>> topic: kafka_topic_A    partition: 178    offsets: 1020632536 to
>>> 1029178618
>>> topic: kafka_topic_A    partition: 52    offsets: 1020842987 to
>>> 1029393669
>>> topic: kafka_topic_A    partition: 5    offsets: 24719725869 to
>>> 24728274543
>>> topic: kafka_topic_A    partition: 63    offsets: 1020887251 to
>>> 1029437144
>>> topic: kafka_topic_B    partition: 36    offsets: 4800982281 to
>>> 4811501000
>>> topic: kafka_topic_A    partition: 11    offsets: 24729694196 to
>>> 24738244559
>>> topic: kafka_topic_A    partition: 69    offsets: 1020732826 to
>>> 1029275514
>>> topic: kafka_topic_A    partition: 89    offsets: 1020642269 to
>>> 1029187888
>>> topic: kafka_topic_B    partition: 11    offsets: 4808218495 to
>>> 4818733612
>>> topic: kafka_topic_B    partition: 25    offsets: 4798933350 to
>>> 4809448450
>>> topic: kafka_topic_A    partition: 96    offsets: 1020846117 to
>>> 1029393750
>>> topic: kafka_topic_B    partition: 10    offsets: 4803818779 to
>>> 4814337498
>>> topic: kafka_topic_A    partition: 37    offsets: 24739837165 to
>>> 24748391468
>>> topic: kafka_topic_B    partition: 32    offsets: 4810693793 to
>>> 4821217501
>>> topic: kafka_topic_A    partition: 134    offsets: 1020747722 to
>>> 1029296407
>>> topic: kafka_topic_A    partition: 13    offsets: 24734355357 to
>>> 24742905825
>>> topic: kafka_topic_A    partition: 19    offsets: 24732775735 to
>>> 24741322331
>>> topic: kafka_topic_A    partition: 229    offsets: 1020798266 to
>>> 1029347927
>>> topic: kafka_topic_A    partition: 91    offsets: 1020974276 to
>>> 1029525120
>>> topic: kafka_topic_A    partition: 64    offsets: 1020980318 to
>>> 1029530189
>>> topic: kafka_topic_A    partition: 34    offsets: 24723495628 to
>>> 24732054835
>>> topic: kafka_topic_A    partition: 4    offsets: 24727632125 to
>>> 24736184191
>>> topic: kafka_topic_A    partition: 175    offsets: 1020915534 to
>>> 1029464464
>>> topic: kafka_topic_A    partition: 53    offsets: 1020704573 to
>>> 1029254608
>>> topic: kafka_topic_A    partition: 143    offsets: 1020772985 to
>>> 1029322428
>>> topic: kafka_topic_A    partition: 118    offsets: 1020778666 to
>>> 1029331391
>>> topic: kafka_topic_A    partition: 249    offsets: 1020963635 to
>>> 1029516291
>>> topic: kafka_topic_A    partition: 3    offsets: 24721520599 to
>>> 24730075720
>>> topic: kafka_topic_A    partition: 184    offsets: 1020775444 to
>>> 1029326031
>>> topic: kafka_topic_A    partition: 225    offsets: 1020933583 to
>>> 1029483635
>>> topic: kafka_topic_A    partition: 188    offsets: 1020647943 to
>>> 1029198446
>>> topic: kafka_topic_A    partition: 94    offsets: 1020730941 to
>>> 1029278716
>>> topic: kafka_topic_A    partition: 213    offsets: 1020762226 to
>>> 1029311435
>>> topic: kafka_topic_A    partition: 151    offsets: 1020844374 to
>>> 1029395379
>>> topic: kafka_topic_A    partition: 125    offsets: 1020760525 to
>>> 1029306817
>>> topic: kafka_topic_A    partition: 139    offsets: 1020830596 to
>>> 1029382287
>>> topic: kafka_topic_A    partition: 223    offsets: 1020851931 to
>>> 1029406373
>>> topic: kafka_topic_A    partition: 79    offsets: 1020569596 to
>>> 1029117673
>>> topic: kafka_topic_B    partition: 41    offsets: 4802503055 to
>>> 4813020137
>>> topic: kafka_topic_A    partition: 157    offsets: 1020773259 to
>>> 1029323214
>>> topic: kafka_topic_B    partition: 43    offsets: 4807530119 to
>>> 4818051823
>>> topic: kafka_topic_B    partition: 9    offsets: 4801124375 to 4811641360
>>> topic: kafka_topic_A    partition: 121    offsets: 1020716814 to
>>> 1029262616
>>> topic: kafka_topic_A    partition: 78    offsets: 1020757202 to
>>> 1029307937
>>> topic: kafka_topic_A    partition: 43    offsets: 24728638290 to
>>> 24737193015
>>> topic: kafka_topic_A    partition: 113    offsets: 1020840637 to
>>> 1029386523
>>> topic: kafka_topic_A    partition: 219    offsets: 1020867425 to
>>> 1029414624
>>> topic: kafka_topic_A    partition: 17    offsets: 24719427351 to
>>> 24727972412
>>> topic: kafka_topic_A    partition: 156    offsets: 1020795237 to
>>> 1029341015
>>> topic: kafka_topic_A    partition: 70    offsets: 1020706495 to
>>> 1029254472
>>> topic: kafka_topic_A    partition: 61    offsets: 1021026951 to
>>> 1029582817
>>> topic: kafka_topic_A    partition: 190    offsets: 1020963590 to
>>> 1029516326
>>> topic: kafka_topic_A    partition: 29    offsets: 24722142896 to
>>> 24730694155
>>> topic: kafka_topic_A    partition: 207    offsets: 1020639874 to
>>> 1029187494
>>> topic: kafka_topic_A    partition: 177    offsets: 1020685282 to
>>> 1029233121
>>> topic: kafka_topic_A    partition: 160    offsets: 1020789969 to
>>> 1029337510
>>> topic: kafka_topic_A    partition: 102    offsets: 1020963819 to
>>> 1029516283
>>> topic: kafka_topic_B    partition: 20    offsets: 4801028715 to
>>> 4811550727
>>> topic: kafka_topic_B    partition: 13    offsets: 4797383641 to
>>> 4807902682
>>> topic: kafka_topic_A    partition: 128    offsets: 1020662803 to
>>> 1029211499
>>> topic: kafka_topic_A    partition: 215    offsets: 1020837321 to
>>> 1029389104
>>> topic: kafka_topic_A    partition: 240    offsets: 1021021049 to
>>> 1029572788
>>> topic: kafka_topic_A    partition: 56    offsets: 1020941937 to
>>> 1029496916
>>> topic: kafka_topic_A    partition: 147    offsets: 1020755896 to
>>> 1029303241
>>> topic: kafka_topic_A    partition: 112    offsets: 1020892430 to
>>> 1029441614
>>> topic: kafka_topic_A    partition: 45    offsets: 24716641715 to
>>> 24725192614
>>> topic: kafka_topic_A    partition: 68    offsets: 1020893444 to
>>> 1029446558
>>> topic: kafka_topic_A    partition: 77    offsets: 1020868499 to
>>> 1029417133
>>> topic: kafka_topic_B    partition: 28    offsets: 4805914153 to
>>> 4816430998
>>> topic: kafka_topic_A    partition: 161    offsets: 1020902852 to
>>> 1029456951
>>> topic: kafka_topic_A    partition: 186    offsets: 1020775276 to
>>> 1029328133
>>> topic: kafka_topic_B    partition: 14    offsets: 4796300859 to
>>> 4806817229
>>> topic: kafka_topic_A    partition: 44    offsets: 24731321741 to
>>> 24739866858
>>> topic: kafka_topic_A    partition: 47    offsets: 24726144390 to
>>> 24734696944
>>> topic: kafka_topic_A    partition: 86    offsets: 1020778038 to
>>> 1029327512
>>> topic: kafka_topic_A    partition: 46    offsets: 24721377928 to
>>> 24729930715
>>> topic: kafka_topic_A    partition: 200    offsets: 1020776353 to
>>> 1029328471
>>> topic: kafka_topic_A    partition: 132    offsets: 1020794282 to
>>> 1029343725
>>> topic: kafka_topic_A    partition: 100    offsets: 1020931503 to
>>> 1029480173
>>> topic: kafka_topic_A    partition: 212    offsets: 1020752903 to
>>> 1029303842
>>> topic: kafka_topic_A    partition: 193    offsets: 1020799750 to
>>> 1029348032
>>> topic: kafka_topic_A    partition: 239    offsets: 1020740938 to
>>> 1029296021
>>> topic: kafka_topic_A    partition: 242    offsets: 1021023598 to
>>> 1029575545
>>> topic: kafka_topic_B    partition: 40    offsets: 4801026818 to
>>> 4811537565
>>> topic: kafka_topic_B    partition: 12    offsets: 4798606447 to
>>> 4809123173
>>> topic: kafka_topic_A    partition: 18    offsets: 24725102864 to
>>> 24733647562
>>> topic: kafka_topic_A    partition: 33    offsets: 24729427865 to
>>> 24737975446
>>> topic: kafka_topic_A    partition: 16    offsets: 24725461165 to
>>> 24734010070
>>> topic: kafka_topic_A    partition: 234    offsets: 1020679052 to
>>> 1029226903
>>> topic: kafka_topic_A    partition: 127    offsets: 1020876420 to
>>> 1029425258
>>> topic: kafka_topic_A    partition: 173    offsets: 1020875774 to
>>> 1029427802
>>> topic: kafka_topic_A    partition: 174    offsets: 1020764367 to
>>> 1029311197
>>> topic: kafka_topic_A    partition: 60    offsets: 1020729422 to
>>> 1029280479
>>> topic: kafka_topic_A    partition: 164    offsets: 1020895388 to
>>> 1029447072
>>> topic: kafka_topic_B    partition: 3    offsets: 4801150811 to 4811667621
>>> topic: kafka_topic_A    partition: 76    offsets: 1020872633 to
>>> 1029425200
>>> topic: kafka_topic_A    partition: 2    offsets: 24720552836 to
>>> 24729103435
>>> topic: kafka_topic_A    partition: 31    offsets: 24724971328 to
>>> 24733525699
>>> topic: kafka_topic_A    partition: 180    offsets: 1020790913 to
>>> 1029342607
>>> topic: kafka_topic_A    partition: 7    offsets: 24722917305 to
>>> 24731461090
>>> topic: kafka_topic_A    partition: 0    offsets: 24715978894 to
>>> 24724533838
>>> topic: kafka_topic_B    partition: 6    offsets: 4801685031 to 4812197203
>>> topic: kafka_topic_A    partition: 111    offsets: 1020777248 to
>>> 1029320002
>>> topic: kafka_topic_A    partition: 214    offsets: 1020847267 to
>>> 1029397260
>>> topic: kafka_topic_A    partition: 183    offsets: 1020829424 to
>>> 1029374366
>>> topic: kafka_topic_A    partition: 247    offsets: 1020951407 to
>>> 1029501748
>>> topic: kafka_topic_A    partition: 35    offsets: 24724710806 to
>>> 24733257282
>>> topic: kafka_topic_B    partition: 2    offsets: 4799162386 to
>>> 4809677022
>>> topic: kafka_topic_B    partition: 23    offsets: 4806523148 to
>>> 4817037826
>>> topic: kafka_topic_A    partition: 84    offsets: 1021016106 to
>>> 1029568619
>>> topic: kafka_topic_B    partition: 31    offsets: 4807475059 to
>>> 4817992907
>>> topic: kafka_topic_A    partition: 15    offsets: 24722975566 to
>>> 24731525636
>>> topic: kafka_topic_A    partition: 238    offsets: 1020838617 to
>>> 1029388674
>>> topic: kafka_topic_A    partition: 217    offsets: 1020963813 to
>>> 1029516908
>>> topic: kafka_topic_A    partition: 141    offsets: 1020928927 to
>>> 1029480391
>>> topic: kafka_topic_B    partition: 21    offsets: 4799274035 to
>>> 4809790430
>>> topic: kafka_topic_A    partition: 142    offsets: 1020859803 to
>>> 1029410671
>>> topic: kafka_topic_A    partition: 26    offsets: 24716858647 to
>>> 24725403869
>>> topic: kafka_topic_A    partition: 75    offsets: 1020875615 to
>>> 1029425108
>>> topic: kafka_topic_A    partition: 88    offsets: 1020636598 to
>>> 1029181677
>>> topic: kafka_topic_A    partition: 55    offsets: 1020981245 to
>>> 1029532042
>>> topic: kafka_topic_B    partition: 26    offsets: 4802386319 to
>>> 4812903171
>>> topic: kafka_topic_A    partition: 176    offsets: 1020927564 to
>>> 1029478273
>>> topic: kafka_topic_A    partition: 246    offsets: 1020902960 to
>>> 1029456226
>>> topic: kafka_topic_A    partition: 237    offsets: 1020879351 to
>>> 1029428560
>>> topic: kafka_topic_A    partition: 124    offsets: 1020844750 to
>>> 1029398619
>>> topic: kafka_topic_A    partition: 216    offsets: 1020606507 to
>>> 1029155109
>>> topic: kafka_topic_A    partition: 32    offsets: 24727599739 to
>>> 24736149128
>>> topic: kafka_topic_A    partition: 25    offsets: 24740711757 to
>>> 24749263320
>>> topic: kafka_topic_A    partition: 197    offsets: 1021032158 to
>>> 1029587829
>>> topic: kafka_topic_B    partition: 44    offsets: 4810511791 to
>>> 4821029704
>>> topic: kafka_topic_A    partition: 95    offsets: 1020733833 to
>>> 1029283829
>>> topic: kafka_topic_A    partition: 12    offsets: 24723998129 to
>>> 24732553534
>>> topic: kafka_topic_A    partition: 109    offsets: 1020895980 to
>>> 1029446212
>>> topic: kafka_topic_B    partition: 22    offsets: 4801811942 to
>>> 4812330157
>>> topic: kafka_topic_A    partition: 135    offsets: 1020523998 to
>>> 1029067367
>>> topic: kafka_topic_B    partition: 48    offsets: 4805322090 to
>>> 4815838865
>>> topic: kafka_topic_A    partition: 74    offsets: 1020819147 to
>>> 1029369936
>>> topic: kafka_topic_A    partition: 230    offsets: 1020784136 to
>>> 1029333313
>>> topic: kafka_topic_A    partition: 103    offsets: 1020921485 to
>>> 1029473542
>>> topic: kafka_topic_B    partition: 34    offsets: 4801025503 to
>>> 4811545042
>>> topic: kafka_topic_A    partition: 115    offsets: 1020600722 to
>>> 1029148541
>>> topic: kafka_topic_A    partition: 152    offsets: 1020677041 to
>>> 1029226178
>>> topic: kafka_topic_A    partition: 158    offsets: 1020735842 to
>>> 1029285162
>>> topic: kafka_topic_A    partition: 210    offsets: 1020838912 to
>>> 1029389328
>>> topic: kafka_topic_A    partition: 123    offsets: 1020888750 to
>>> 1029442669
>>> topic: kafka_topic_A    partition: 49    offsets: 24733516034 to
>>> 24742064144
>>> topic: kafka_topic_B    partition: 39    offsets: 4806601961 to
>>> 4817119869
>>> topic: kafka_topic_A    partition: 114    offsets: 1020945219 to
>>> 1029496002
>>> topic: kafka_topic_A    partition: 65    offsets: 1020714711 to
>>> 1029267579
>>> topic: kafka_topic_A    partition: 98    offsets: 1020581086 to
>>> 1029126420
>>> topic: kafka_topic_B    partition: 33    offsets: 4802443872 to
>>> 4812950776
>>> topic: kafka_topic_A    partition: 73    offsets: 1020908814 to
>>> 1029459329
>>> topic: kafka_topic_A    partition: 14    offsets: 24720549899 to
>>> 24729100604
>>> topic: kafka_topic_A    partition: 106    offsets: 1020832194 to
>>> 1029381879
>>> topic: kafka_topic_B    partition: 46    offsets: 4805759222 to
>>> 4816272314
>>> topic: kafka_topic_A    partition: 130    offsets: 1020729244 to
>>> 1029276701
>>> topic: kafka_topic_A    partition: 166    offsets: 1020939071 to
>>> 1029489456
>>> topic: kafka_topic_A    partition: 104    offsets: 1020771720 to
>>> 1029318470
>>> topic: kafka_topic_A    partition: 224    offsets: 1021062976 to
>>> 1029618193
>>> topic: kafka_topic_B    partition: 0    offsets: 4805841603 to
>>> 4816356537
>>> topic: kafka_topic_A    partition: 39    offsets: 24733836602 to
>>> 24742385677
>>> topic: kafka_topic_A    partition: 202    offsets: 1020738496 to
>>> 1029289191
>>> topic: kafka_topic_A    partition: 62    offsets: 1020767369 to
>>> 1029310260
>>> topic: kafka_topic_A    partition: 54    offsets: 1020872832 to
>>> 1029424418
>>> topic: kafka_topic_A    partition: 155    offsets: 1020939790 to
>>> 1029491266
>>> topic: kafka_topic_A    partition: 57    offsets: 1020926473 to
>>> 1029478170
>>> topic: kafka_topic_A    partition: 10    offsets: 24722360402 to
>>> 24730916736
>>> topic: kafka_topic_A    partition: 227    offsets: 1020628274 to
>>> 1029175330
>>> topic: kafka_topic_A    partition: 205    offsets: 1020886863 to
>>> 1029438420
>>> topic: kafka_topic_A    partition: 9    offsets: 24730599499 to
>>> 24739147248
>>> topic: kafka_topic_A    partition: 218    offsets: 1020694139 to
>>> 1029244205
>>> topic: kafka_topic_A    partition: 81    offsets: 1020865158 to
>>> 1029417909
>>> topic: kafka_topic_A    partition: 99    offsets: 1020829095 to
>>> 1029378716
>>> topic: kafka_topic_A    partition: 144    offsets: 1020836880 to
>>> 1029390098
>>> topic: kafka_topic_A    partition: 80    offsets: 1020632760 to
>>> 1029181116
>>> topic: kafka_topic_A    partition: 185    offsets: 1020777167 to
>>> 1029326135
>>> topic: kafka_topic_A    partition: 137    offsets: 1020783286 to
>>> 1029336240
>>> topic: kafka_topic_A    partition: 145    offsets: 1020807427 to
>>> 1029353122
>>> topic: kafka_topic_A    partition: 122    offsets: 1020914744 to
>>> 1029465920
>>> topic: kafka_topic_A    partition: 133    offsets: 1020818950 to
>>> 1029369827
>>> topic: kafka_topic_A    partition: 71    offsets: 1020604295 to
>>> 1029151699
>>> topic: kafka_topic_A    partition: 82    offsets: 1020925125 to
>>> 1029478280
>>> topic: kafka_topic_A    partition: 87    offsets: 1020857237 to
>>> 1029406722
>>> topic: kafka_topic_A    partition: 201    offsets: 1020709307 to
>>> 1029260228
>>> topic: kafka_topic_A    partition: 28    offsets: 24728200955 to
>>> 24736749015
>>> topic: kafka_topic_A    partition: 41    offsets: 24729533353 to
>>> 24738085917
>>> topic: kafka_topic_A    partition: 170    offsets: 1020668802 to
>>> 1029219950
>>> topic: kafka_topic_A    partition: 187    offsets: 1020581810 to
>>> 1029129601
>>> topic: kafka_topic_B    partition: 29    offsets: 4803280139 to
>>> 4813797539
>>> topic: kafka_topic_A    partition: 92    offsets: 1020662671 to
>>> 1029214523
>>> topic: kafka_topic_A    partition: 231    offsets: 1020772888 to
>>> 1029320782
>>> topic: kafka_topic_A    partition: 241    offsets: 1020649136 to
>>> 1029195109
>>> topic: kafka_topic_A    partition: 192    offsets: 1020839092 to
>>> 1029389989
>>> topic: kafka_topic_A    partition: 8    offsets: 24732792451 to
>>> 24741339710
>>> topic: kafka_topic_A    partition: 131    offsets: 1020886007 to
>>> 1029433501
>>> topic: kafka_topic_A    partition: 162    offsets: 1020706400 to
>>> 1029251727
>>> topic: kafka_topic_A    partition: 126    offsets: 1020828002 to
>>> 1029377579
>>> topic: kafka_topic_A    partition: 228    offsets: 1020824139 to
>>> 1029371645
>>> topic: kafka_topic_A    partition: 167    offsets: 1020746310 to
>>> 1029296452
>>> topic: kafka_topic_B    partition: 30    offsets: 4795764234 to
>>> 4806277616
>>> topic: kafka_topic_A    partition: 221    offsets: 1020618597 to
>>> 1029166130
>>> topic: kafka_topic_A    partition: 206    offsets: 1020972294 to
>>> 1029522361
>>> topic: kafka_topic_A    partition: 245    offsets: 1020859155 to
>>> 1029409690
>>> topic: kafka_topic_A    partition: 148    offsets: 1020689094 to
>>> 1029234764
>>> topic: kafka_topic_A    partition: 171    offsets: 1020893286 to
>>> 1029448085
>>> topic: kafka_topic_A    partition: 20    offsets: 24727739340 to
>>> 24736287861
>>> topic: kafka_topic_A    partition: 159    offsets: 1020770845 to
>>> 1029316911
>>> topic: kafka_topic_A    partition: 169    offsets: 1020699633 to
>>> 1029253155
>>> topic: kafka_topic_A    partition: 83    offsets: 1020954835 to
>>> 1029507004
>>> topic: kafka_topic_A    partition: 149    offsets: 1020763182 to
>>> 1029312029
>>> topic: kafka_topic_B    partition: 17    offsets: 4798809279 to
>>> 4809328520
>>> topic: kafka_topic_A    partition: 191    offsets: 1020939618 to
>>> 1029492433
>>> topic: kafka_topic_A    partition: 50    offsets: 1020781205 to
>>> 1029327065
>>> topic: kafka_topic_A    partition: 107    offsets: 1020596042 to
>>> 1029143966
>>> topic: kafka_topic_A    partition: 179    offsets: 1020692875 to
>>> 1029239892
>>> topic: kafka_topic_A    partition: 204    offsets: 1020682012 to
>>> 1029229892
>>> topic: kafka_topic_B    partition: 15    offsets: 4797528038 to
>>> 4808038327
>>> topic: kafka_topic_A    partition: 198    offsets: 1020530213 to
>>> 1029075405
>>> topic: kafka_topic_B    partition: 45    offsets: 4803051802 to
>>> 4813564524
>>> topic: kafka_topic_A    partition: 129    offsets: 1020804825 to
>>> 1029355767
>>>
>>>
>>> On Wed, Nov 2, 2016 at 11:21 AM, map reduced <k3t.git.1@gmail.com>
>>> wrote:
>>>
>>>> Yes it does, I checked in the logs. Infact, if you see the first
>>>> screenshot, stream processing was 'stuck' processing those many records for
>>>> quite some time (~ 1hr).
>>>> One thing I noticed is initial batches took (maybe far?) longer than
>>>> the configured batchDuration of 1.5mins, say in case screenshot 2, it took
>>>> 5.8-7.1min and in case 1 it took 3-4 mins.
>>>>
>>>> On Wed, Nov 2, 2016 at 8:43 AM, Cody Koeninger <cody@koeninger.org>
>>>> wrote:
>>>>
>>>>> Does that batch actually have that many records in it (you should be
>>>>> able to see beginning and ending offsets in the logs), or is it an error
in
>>>>> the UI?
>>>>>
>>>>>
>>>>> On Tue, Nov 1, 2016 at 11:59 PM, map reduced <k3t.git.1@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I am using Spark 2.0.0 standalone cluster, regular streaming job
>>>>>> consuming from kafka and writing to http endpoint. I have configuration:
>>>>>> executors 7 cores/executor, maxCores = 84 (so 12 executors)
>>>>>> batchsize - 90 seconds
>>>>>> maxRatePerPartition - 2000
>>>>>> backPressure enabled = true
>>>>>>
>>>>>> My kafka topics have total of 300 partitions, so I am expecting to
be
>>>>>> max 54million records per batch (maxRatePerPartition * batchsize
*
>>>>>> #partitions) - and that's what I am getting. But it turns out that
it can't
>>>>>> process 54million records in 90sec batch, so I am expecting backpressure
to
>>>>>> kick in, but I see something strange there. It reduces batch size
to lesser
>>>>>> # of records, but then suddenly spits out a HUGE batch size of 13
billion
>>>>>> records.
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>> I changed some configuration to see if above was a one off case but
>>>>>> the same issue happened again. Check the below screenshot (huge batch
size
>>>>>> of 14 billion records again!) :
>>>>>>
>>>>>> [image: Inline image 2]
>>>>>>
>>>>>> Is this a bug? Any reasoning you know for this to happen?
>>>>>>
>>>>>> Thanks,
>>>>>> KP
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message