spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: Possibly a memory leak issue in Spark
Date Wed, 22 Sep 2021 04:30:18 GMT
No, that's just info Spark retains about finished jobs and tasks, likely.
You can limit how much is retained if desired with config.

On Tue, Sep 21, 2021, 11:29 PM Kohki Nishio <taroplus@gmail.com> wrote:

> Just following up, it looks like task / stage / job data are not cleaned up
> --
>    6:       7835346     2444627952  org.apache.spark.status.TaskDataWrapper
>  25:       3765152      180727296  org.apache.spark.status.StageDataWrapper
> 88:        232255        9290200  org.apache.spark.status.JobDataWrapper
>
> UI is disabled, not sure why we need to have those data ..
>
> -Kohki
>
>
> On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <taroplus@gmail.com> wrote:
>
>> Hello,
>> I'm seeing possible memory leak behavior in my spark application.
>> According to MAT, it looks like it's related to ElementTrackingStore ..
>>
>> [image: Eclipse_Memory_Analyzer.png]
>>
>> The increase is subtle, so it takes multiple days to actually cause some
>> impact, but I'm wondering if anybody has any idea about what this is about
>> ...  Below is the GC graph, yellow is the level after any GC kicks in.
>>
>> [image: T2_G1GC_-_Grafana.png]
>>
>> Thanks
>> --
>> Kohki Nishio
>>
>
>
> --
> Kohki Nishio
>

Mime
View raw message