spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Tasks are skewed to one executor
Date Sat, 10 Apr 2021 14:42:14 GMT
Hi,

Can you provide a bit more info please?

How are you running this job and what is the streaming framework (kafka,
files etc)?

HTH


Mich


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sat, 10 Apr 2021 at 14:28, AndrĂ¡s Kolbert <kolbertandras@gmail.com>
wrote:

> hi,
>
> I have a streaming job and quite often executors die (due to memory
> errors/ "unable to find location for shuffle etc) during the processing. I
> started digging and found that some of the tasks are concentrated to one
> executor, just as below:
> [image: image.png]
>
> Can this be the reason?
> Should I repartition the underlying data before I execute a groupby on the
> top of it?
>
> Any advice is welcome
>
> Thanks
> Andras
>

Mime
View raw message