spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grega Kešpret <gr...@celtra.com>
Subject spark.task.maxFailures
Date Fri, 29 Nov 2013 10:02:05 GMT
Looking at http://spark.incubator.apache.org/docs/latest/configuration.html
docs says:
Number of individual task failures before giving up on the job. Should be
greater than or equal to 1. Number of allowed retries = this value - 1.

However, looking at the code
https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala#L532

if I set spark.task.maxFailures to 1, this means that job will fail after
task fails for the second time. Shouldn't this line be corrected to if (
numFailures(index) >= MAX_TASK_FAILURES) {
?

I can open a pull request if this is the case.

Thanks,
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer

Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message