spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AndrĂ¡s Kolbert <kolbertand...@gmail.com>
Subject Increase batch interval in case of delay
Date Thu, 01 Jul 2021 11:31:27 GMT
Hi,

I have a spark streaming application which generally able to process the
data within the given time frame. However, in certain hours it starts
increasing that causes a delay.

In my scenario, the number of input records are not linearly increase the
processing time. Hence, ideally I'd like to increase the number of
batches/records that are being processed after a delay reaches a certain
time.

Is there a possibility/settings to do so?

Thanks
Andras


[image: image.png]

Mime
View raw message