spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kohki Nishio <tarop...@gmail.com>
Subject Re: Possibly a memory leak issue in Spark
Date Wed, 22 Sep 2021 04:21:43 GMT
Just following up, it looks like task / stage / job data are not cleaned up
--
   6:       7835346     2444627952  org.apache.spark.status.TaskDataWrapper
 25:       3765152      180727296  org.apache.spark.status.StageDataWrapper
88:        232255        9290200  org.apache.spark.status.JobDataWrapper

UI is disabled, not sure why we need to have those data ..

-Kohki


On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <taroplus@gmail.com> wrote:

> Hello,
> I'm seeing possible memory leak behavior in my spark application.
> According to MAT, it looks like it's related to ElementTrackingStore ..
>
> [image: Eclipse_Memory_Analyzer.png]
>
> The increase is subtle, so it takes multiple days to actually cause some
> impact, but I'm wondering if anybody has any idea about what this is about
> ...  Below is the GC graph, yellow is the level after any GC kicks in.
>
> [image: T2_G1GC_-_Grafana.png]
>
> Thanks
> --
> Kohki Nishio
>


-- 
Kohki Nishio

Mime
View raw message