hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14154) Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
Date Wed, 08 Mar 2017 23:57:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902191#comment-15902191

Aaron Fabbri commented on HADOOP-14154:

Hi [~rajesh.balamohan].  Thanks for the patch.  This is an area we plan to improve (to allow
S3A to avoid round trips to s3 when MetadataStore has full listing).

I think you're confusing the "isAuthoritative" bit on directories with the "allow authoritative"
config parameter for s3a.  I hope some existing integration tests would fail with this patch
to catch this.

The "isAuthoritative" flag is, essentially, a boolean on a DirListingMetadata that is used
by the FS client to tell the MetadataStore (MS) that "this is the full listing for this directory".
  If a client does put() with a DirListingMetadata that has isAuthoritative == true, then
subsequent listChildren() *should* return the listing with isAuthoritative == true.   A MetadataStore
*may* chose not to persist that isAuthoritative bit on the directory listing, though.  This
is what DDB does today: it does not store the isAuthoritative bit on directories.  Thus, S3A
will always check S3 as well as DDB MS because DDB doesn't remember that it was given a full

A MetadataStore *must not* return isAuthoritative==true for listChildren() unless the FS client
previously set that bit on the directory.  The MS only knows that it has the full listing
if the FS client tells it.

The "allow authoritative" config flag for S3A, when true, tells S3A that it may skip the list
operation on S3 if the MS returns a full directory listing (isAuthoritative==true on the DirListingMetadata).
 So that config flag should only be used in S3A.

The correct way to address this issue is to have DDB MS persist the isAuthoritative bit on

Does that help clarify the architecture here?

> Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
> ------------------------------------------------------------------------------
>                 Key: HADOOP-14154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14154
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HADOOP-14154-HADOOP-13345.001.patch, HADOOP-14154-HADOOP-13345.002.patch
> Currently {{DynamoDBMetaStore::listChildren}} does not populate {{isAuthoritative}} flag
when creating {{DirListingMetadata}}. 
> This causes additional S3 lookups even when users have enabled {{fs.s3a.metadatastore.authoritative}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message