spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Biplob Biswas <revolutioni...@gmail.com>
Subject Re: [Spark-Core] Long scheduling delays (1+ hour)
Date Wed, 07 Nov 2018 10:53:38 GMT
Hi,

This has to do with your batch duration and processing time, as a rule, the
batch duration should be lower than the processing time of your data. As I
can see from your screenshots, your batch duration is 10 seconds but your
processing time is more than a minute mostly, this adds up and you will end
up having a lot of scheduling delay.

Maybe see, why does it take 1 min to process 100 records and fix the logic.
Also, I see you have higher number of events which takes some time lower
amount of processing time. Fix the code logic and this should be fixed.

Thanks & Regards
Biplob Biswas


On Wed, Nov 7, 2018 at 11:08 AM bsikander <behroz89@gmail.com> wrote:

> We are facing an issue with very long scheduling delays in Spark (upto 1+
> hours).
> We are using Spark-standalone. The data is being pulled from Kafka.
>
> Any help would be much appreciated.
>
> I have attached the screenshots.
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/1-stats.png>
>
> <http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/4.png>
> <http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/3.png>
> <http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/2.png>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message