spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kay Ousterhout (JIRA)" <>
Subject [jira] [Closed] (SPARK-19502) Remove unnecessary code to re-submit stages in the DAGScheduler
Date Sat, 11 Feb 2017 06:32:41 GMT


Kay Ousterhout closed SPARK-19502.
    Resolution: Not A Problem

This code actually is currently needed to handle cases where a ShuffleMapTask succeeds on
an executor, but that executor was marked as failed (so the task needs to be re-run), as described
in this comment:

> Remove unnecessary code to re-submit stages in the DAGScheduler
> ---------------------------------------------------------------
>                 Key: SPARK-19502
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Scheduler
>    Affects Versions: 1.1.1
>            Reporter: Kay Ousterhout
>            Assignee: Kay Ousterhout
>            Priority: Minor
> There are a [few lines of code in the DAGScheduler](
to re-submit shuffle map stages when some of the tasks fail.  My understanding is that there
should be a 1:1 mapping between pending tasks (which are tasks that haven't completed successfully)
and available output locations, so that code should never be reachable.  Furthermore, the
approach taken by that code (to re-submit an entire stage as a result of task failures) is
not how we handle task failures in a stage (the lower-level scheduler resubmits the individual
tasks) which is what the 5-years-old TODO on that code seems to be implying should be done.
> The big caveat is that there's a bug being fixed in SPARK-19263 that means there is *not*
a 1:1 relationship between pendingTasks and available outputLocations, so that code is serving
as a (buggy) band-aid.  This should be fixed once we resolve SPARK-19263.
> cc [~imranr] [~markhamstra] [] (let me know if any of you see any
reason we actually do need that code)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message