spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AndrĂ¡s Kolbert <>
Subject Increase batch interval in case of delay
Date Thu, 01 Jul 2021 11:31:27 GMT

I have a spark streaming application which generally able to process the
data within the given time frame. However, in certain hours it starts
increasing that causes a delay.

In my scenario, the number of input records are not linearly increase the
processing time. Hence, ideally I'd like to increase the number of
batches/records that are being processed after a delay reaches a certain

Is there a possibility/settings to do so?


[image: image.png]

View raw message