hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16090) S3A Client to add explicit support for versioned stores
Date Tue, 19 Mar 2019 10:45:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795964#comment-16795964

Steve Loughran commented on HADOOP-16090:

I'm dependent on other people's reviews, so a couple-of-days was going to be the bare minimum

* love to see what you've been doing
* assume that I'll be backporting the invoker stuff to branch-2 (but not the current use throughout
* refactoring getFileStatus. Hmm. Popular entry point. Slow enough over long-haul you can
see the logs pausing.

I'm actually sketching out (but yet to publish) something about incremental refactoring of
the S3A code itself: the S3AFileSystem class has got over complex.

Something like

* {{org.apache.hadoop.fs.s3a.impl.S3Core}}:  S3-api level, with state coming in as context,
link to executor pool
* {{org.apache.hadoop.fs.s3a.impl.S3AModel}}: the filesystem model atop the core: S3Guard,
directory trees etc.
* {{S3AFileSytem}}: Hadoop API

Core doesn't call model; model doesn't call FS. 

At the same time, no grand "break all backporting" patch. 

Anyhow, assume getObjectMetadata is the lowest-level, but in the S3AModel it'd be going through
S3Guard to, in future, pick up version info from there as/when collected. If you've not been
keeping an eye on things. input streams are now version aware too.

> S3A Client to add explicit support for versioned stores
> -------------------------------------------------------
>                 Key: HADOOP-16090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16090
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.1
>            Reporter: Dmitri Chmelev
>            Assignee: Steve Loughran
>            Priority: Minor
> The fix to avoid calls to getFileStatus() for each path component in deleteUnnecessaryFakeDirectories()
(HADOOP-13164) results in accumulation of delete markers in versioned S3 buckets. The above
patch replaced getFileStatus() checks with a single batch delete request formed by generating
all ancestor keys formed from a given path. Since the delete request is not checking for existence
of fake directories, it will create a delete marker for every path component that did not
exist (or was previously deleted). Note that issuing a DELETE request without specifying
a version ID will always create a new delete marker, even if one already exists ([AWS S3
Developer Guide|https://docs.aws.amazon.com/AmazonS3/latest/dev/RemDelMarker.html])
> Since deleteUnnecessaryFakeDirectories() is called as a callback on successful writes
and on renames, delete markers accumulate rather quickly and their rate of accumulation is
inversely proportional to the depth of the path. In other words, directories closer to the
root will have more delete markers than the leaves.
> This behavior negatively impacts performance of getFileStatus() operation when it has
to issue listObjects() request (especially v1) as the delete markers have to be examined when
the request searches for first current non-deleted version of an object following a given
> I did a quick comparison against 3.x and the issue is still present: [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message