spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peng Cheng <>
Subject Re: How to enable fault-tolerance?
Date Mon, 09 Jun 2014 18:24:43 GMT
Thanks a lot! That's very responsive, somebody definitely has 
encountered the same problem before, and added two hidden modes in 

(from SparkContext.scala: line1431)

    // Regular expression for local[N, maxRetries], used in tests with 
failing tasks
    // Regular expression for simulating a Spark cluster of [N, cores, 
memory] locally

Unfortunately they never got pushed into the documentation, and you got 
config parameters scattered in two different places (masterURL and 
I'm thinking of adding a new config parameter 
$spark.task.maxLocalFailures to override 1, how do you think?

Thanks again buddy.

Yours Peng

On Mon 09 Jun 2014 01:33:45 PM EDT, Aaron Davidson wrote:
> Looks like your problem is local mode:
> For some reason, someone decided not to do retries when running in
> local mode. Not exactly sure why, feel free to submit a JIRA on this.
> On Mon, Jun 9, 2014 at 8:59 AM, Peng Cheng <
> <>> wrote:
>     I speculate that Spark will only retry on exceptions that are
>     registered with
>     TaskSetScheduler, so a definitely-will-fail task will fail quickly
>     without
>     taking more resources. However I haven't found any documentation
>     or web page
>     on it
>     --
>     View this message in context:
>     Sent from the Apache Spark User List mailing list archive at

View raw message