spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominique De Vito (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application
Date Sat, 06 Jul 2019 15:51:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879711#comment-16879711
] 

Dominique De Vito commented on SPARK-19977:
-------------------------------------------

Same pb, but with Spark 2.0.2 (from DataStax Enterprise Edition 5.1.x).

My streaming app reads from Kafka.

During a period of time without ANY received data (0 msg read from any partition), the scheduling
delay is very close to 0 ms. Then, while there is still no msg read, the scheduling delay
increases up to 4 sec, then 7 sec, then 12 sec....

So far, I have not crafted a way to reproduce the bug...

[~rayqiu] : have you met again the bug ? or has it disappeared while upgrading ? Thanks.

 

 

> Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to
30 seconds in Spark Streaming application
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19977
>                 URL: https://issues.apache.org/jira/browse/SPARK-19977
>             Project: Spark
>          Issue Type: Bug
>          Components: Scheduler
>    Affects Versions: 2.1.0
>            Reporter: Ray Qiu
>            Priority: Major
>
> Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to
30+ seconds in a Spark Streaming application, where multiple Kafka direct streams are processed.
 These kafka streams are processed separately (not combined via union).  
> It causes the task processing time to increase greatly and eventually stops working.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message