hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
Date Tue, 07 Mar 2017 03:55:33 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898693#comment-15898693

Aaron Fabbri commented on HADOOP-13914:

Thanks for the review [~liuml07].  Great comments..

In DynamoDBMetadataStore#get(), we should check wantEmptyDirectoryFlag

Good catch.  Next patch will skip that extra work if the caller doesn't care.  I meant to
do that but forgot.

The isAuthoritative is false and we have children, we confidently trust the child's existence
though it's not authoritative (fully cached). Seems reasonable.

Yep..  And when we add support to DDB for tracking isAuthoritative, we may also be able to
answer true if {{authoritative && empty}}.

{quote}To simplify the if clause as if (!allowMissing()){quote}

That logic is intentional:  {{allowMissing()}} means a MetadataStore may discard contents.
 E.g. LocalMetadataStore discards entries based on LRU or (eventually) TTL timeout.  The case
also covers NullMetadataStore which, legally, discards everything.

Even if the MetadataStore is allowed to discard results, if it does return results, we *still*
run validation for the test to make sure the result is correct.  So, to summarize, allowMissing()
means "test shouldn't fail if an entry is missing" but otherwise, all validation should occur.

> s3guard: improve S3AFileStatus#isEmptyDirectory handling
> --------------------------------------------------------
>                 Key: HADOOP-13914
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13914
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13914-HADOOP-13345.000.patch, HADOOP-13914-HADOOP-13345.002.patch,
HADOOP-13914-HADOOP-13345.003.patch, HADOOP-13914-HADOOP-13345.004.patch, HADOOP-13914-HADOOP-13345.005.patch,
s3guard-empty-dirs.md, test-only-HADOOP-13914.patch
> As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag stored in
S3AFileStatus is missing from DynamoDBMetadataStore.
> The approach taken by LocalMetadataStore is not suitable for the DynamoDB implementation,
and also sacrifices good code separation to minimize S3AFileSystem changes pre-merge to trunk.
> I will attach a design doc that attempts to clearly explain the problem and preferred
solution.  I suggest we do this work after merging the HADOOP-13345 branch to trunk, but am
open to suggestions.
> I can also attach a patch of a integration test that exercises the missing case and demonstrates
a failure with DynamoDBMetadataStore.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message