spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Beabes <mailinglist...@gmail.com>
Subject Re: Only one Active task in Spark Structured Streaming application
Date Thu, 21 Jan 2021 17:18:40 GMT
Yes. For this particular use case the state size could be big but I doubt
if there's a leak. Maybe adding more memory would help.

On Thu, Jan 21, 2021 at 5:55 PM Sean Owen <srowen@gmail.com> wrote:

> Is your app accumulating a lot of streaming state? that's one reason
> something could slow down after a long time. Some memory leak in your app
> putting GC/memory pressure on the JVM, etc too.
>
> On Thu, Jan 21, 2021 at 5:13 AM Eric Beabes <mailinglists19@gmail.com>
> wrote:
>
>> Hello,
>>
>> My Spark Structured Streaming application was performing well for quite
>> some time but all of a sudden from today it has slowed down. I noticed in
>> the Spark UI that the 'No. of Active Tasks' is 1 even though 64 Cores are
>> available. (Please see the attached image).
>>
>> I don't believe there's any data skew issue related to partitioning of
>> data. What could be the reason for this? Please advise. Thanks.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message