hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
Date Wed, 11 Jan 2017 14:11:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818440#comment-15818440

Steve Loughran commented on HADOOP-13786:

Multipart algorithm summarised by Thomas Demoor on HADOOP-13912

bq. Object stores do not have an efficient rename operation, which is used by the Hadoop FileOutputCommitter
to atomically promote the "winning" attempt out of the multiple (speculative) attempts to
the final path. These slow job commits are one of the main friction points when using object
stores in Hadoop.There have been quite some attempts at resolving this: HADOOP-9565Link, Apache
Spark DirectOutputCommitters, ... but they have proven not to be robust in face of adversity
(network partitions, ...).
bq. The current ticket proposes to do the atomic commit by using the S3 Multipart API, which
allows multiple concurrent uploads on the same objectname, each in its own "temporary space,
identified by the UploadId which is returned as a response to InitiateMultipartUpload. Every
attempt writes directly to the final outputPath. Data is uploaded using Put Part and as a
response an ETag for the part is returned and stored. The CompleteMultipartUpload is postponed.
Instead, we persist the UploadId (using a _temporary subdir or elsewhere) and the ETags. When
a certain "job" wins CompleteMultipartUpload is called for each of its files using the proper
list of Part ETags.
bq. Completing a MultipartUpload is a metadata only operation (internally in S3) and is thus
orders of magnitude faster than the rename-based approach which moves all the data.

> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>                 Key: HADOOP-13786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13786
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-HADOOP-13345-001.patch
> A goal of this code is "support O(1) commits to S3 repositories in the presence of failures".
Implement it, including whatever is needed to demonstrate the correctness of the algorithm.
(that is, assuming that s3guard provides a consistent view of the presence/absence of blobs,
show that we can commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie.
not visible until the close()), if we need to use that to allow us to abort commit operations.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message