spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Commented] (SPARK-5216) Spark Ui should report estimated time remaining for each stage.
Date Fri, 16 Jan 2015 06:22:34 GMT


Patrick Wendell commented on SPARK-5216:

This has been proposed before, but in the past we decided not to do it. Trying to extrapolate
the finish time of a stage accurately is basically impossible since in many workloads stragglers
dominate the total response time. The conclusion was that it was better to give no estimate
rather than one which is likely to be misleading. 

> Spark Ui should report estimated time remaining for each stage.
> ---------------------------------------------------------------
>                 Key: SPARK-5216
>                 URL:
>             Project: Spark
>          Issue Type: Wish
>          Components: Spark Core, Web UI
>    Affects Versions: 1.3.0
>            Reporter: Prashant Sharma
>            Assignee: Prashant Sharma
> Per stage feedback on estimated remaining time can help user get a grasp on how much
time the job is going to take. This will only require changes on the UI/JobProgressListener
side of code since we already have most of the information needed. 
> In the initial cut, plan is to estimate time based on statistics of running job i.e.
average time taken by each task and number of task per stage. This will makes sense when jobs
are long. And then if this makes sense, then more heuristics can be added like projected time
saved if the rdd is cached and so on. 
> More precise details will come as this evolves. In the meantime thoughts on alternate
ways and suggestion on usefulness are welcome.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message