spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: Increase batch interval in case of delay
Date Thu, 01 Jul 2021 11:45:45 GMT
Wouldn't this happen naturally? the large batches would just take a longer
time to complete already.

On Thu, Jul 1, 2021 at 6:32 AM AndrĂ¡s Kolbert <kolbertandras@gmail.com>
wrote:

> Hi,
>
> I have a spark streaming application which generally able to process the
> data within the given time frame. However, in certain hours it starts
> increasing that causes a delay.
>
> In my scenario, the number of input records are not linearly increase the
> processing time. Hence, ideally I'd like to increase the number of
> batches/records that are being processed after a delay reaches a certain
> time.
>
> Is there a possibility/settings to do so?
>
> Thanks
> Andras
>
>
> [image: image.png]
>

Mime
View raw message