spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luca Canali (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed
Date Thu, 30 Aug 2018 19:45:00 GMT
Luca Canali created SPARK-25285:
-----------------------------------

             Summary: Add executor task metrics to track the number of tasks started and of
tasks successfully completed
                 Key: SPARK-25285
                 URL: https://issues.apache.org/jira/browse/SPARK-25285
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Luca Canali


The motivation for these additional metrics is to help in troubleshooting situations when
tasks fail, are killed and/or restarted. Currently available metrics include executor threadpool
metrics for task completed and for active tasks. The addition of threadpool tasStarted metric
will allow for example to collect info on the (approximate) number of failed tasks by computing
the difference thread started – (active threads + completed tasks and/or successfully completed
tasks).

The proposed metric successfulTasks is also intended for this type of troubleshooting. The
difference between  successfulTasks and threadpool.completeTasks, is that the latter is a
(dropwizard library) gauge taken from the threadpool, while the former is a (dropwizard) counter
computed in the [[Executor]] class, when a task successfully completes, together with several
other task metrics counters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message