spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yana Kadiyska <yana.kadiy...@gmail.com>
Subject Help understanding spark.task.maxFailures
Date Mon, 30 Jun 2014 19:10:33 GMT
Hi community, this one should be an easy one:

I have left spark.task.maxFailures to it's default (which should be
4). I see a job that shows the following statistics for Tasks:
Succeeded/Total

7109/819 (1 failed)

So there were 819 tasks to start with. I have 2 executors in that
cluster. From Spark docs it says spark.task.maxFailures is the number
of times to try a task before a job is given up. So I was imagining
that 819*4 (i.e. 3276) would be the max number to ever see in the
succeeded (accounting for retries on every possibly task). even that
3276*2 (6552, if it's per task per executor) does not account for 7109
successfull tasks.

Could anyone help explain why I'm seeing such high number of succeeded tasks?

Mime
View raw message