hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Understanding task commit/abort protocol
Date Fri, 03 Feb 2017 14:41:48 GMT

In HADOOP-13786 I'm adding a new committer, one which writes to S3 without doing renames.
It does this by submitting all the data to S3 targeted at the final destination, but doesn't
send the POST needed to materialize it until the tasks commits. Abort the task and it cancels
these pending commits.

this algorithm should be robust provided that only one attempt for a task is committed, which
comes down to

1.  Only those tasks which have succeeded are committed
2   those tasks which have not succeeded have their pending writes aborted

Which is where I now have a question. In the class org.apache.hadoop.mapred.Task, OutputCommitter.commitTask()
is called when a task is pre-empted:

  public void done(TaskUmbilicalProtocol umbilical,
                   TaskReporter reporter
                   ) throws IOException, InterruptedException {
    if (taskStatus.getRunState() == TaskStatus.State.PREEMPTED ) {
      // If we are preempted, do no output promotion; signal done and exit
      committer.commitTask(taskContext);         / * HERE */
      umbilical.preempted(taskId, taskStatus);

That's despite the line above saying "do no output promotion", and, judging by its place in
the code, looking like it's the handler for task preempted state.

Shouldn't it be doing a task abort here? 

I suspect the sole reason this hasn't shown up as a problem before is that this is the sole
use of TaskStatus.State.PREEMPTED in the hadoop code: this particular codepath is never executed.
In which case, culling it may be correct option.



To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

View raw message