spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Updated] (SPARK-1272) Don't fail job if some local directories are buggy
Date Sun, 30 Mar 2014 04:13:25 GMT


Patrick Wendell updated SPARK-1272:

    Reporter: Patrick Wendell  (was: Patrick Cogan)

> Don't fail job if some local directories are buggy
> --------------------------------------------------
>                 Key: SPARK-1272
>                 URL:
>             Project: Apache Spark
>          Issue Type: Improvement
>          Components: Shuffle
>            Reporter: Patrick Wendell
>            Assignee: OuyangJin
>             Fix For: 1.0.0
> If Spark cannot create shuffle directories inside of a local directory it might make
sense to just log an error and continue, provided that at least one valid shuffle directory
exists. Otherwise if a single disk is wonky the entire job can fail.
> The down side is that this might mask failures if the person actually misconfigures the
local directories to point to the wrong disk(s).

This message was sent by Atlassian JIRA

View raw message