hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13760) S3Guard: add delete tracking
Date Wed, 31 May 2017 15:16:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031323#comment-16031323
] 

Steve Loughran commented on HADOOP-13760:
-----------------------------------------

just catching up with this. Core code looks ok, but there are some things I'd like tuned

# Why not make use of switch() statements in the S3A FS in {{innerMkdirs}}
# delete() javadocs provide more detail on what's happening. I'd prefer that this isn't in
there, as we describe what's going on in the FS spec documents; that is the behaviour which
is required, and there shouldn't be any need to declare anything different

bq. Except in the special case of the root directory, if this API call
completed successfully then there is nothing at the end of the path.
That is: the outcome is desired. The return flag simply tells the caller
whether or not any change was made to the state of the filesystem.

The error message raised if a parent element is a file has changed and no longer correct

Before: lists the path element at fault
{code}
 throw new FileAlreadyExistsException(String.format(
                "Can't make directory for path '%s' since it is a file.",
                fPart));
{code}
after: lists the destination path, and doesn't declare that there is a parent dir that is
a file
{code}
throw new FileAlreadyExistsException("Path is a file: " + f);
{code}


if you tried to {{mkdirs("/a/b.txt/c/d.txt")}} then before the error would be "Can't make
directory for path '/a/b.txt' since it is a file.". Now it it will be "Path is a file: /a/b.txt/c/d.txt",
which isn't correct

Can I get these looked at? Thanks


> S3Guard: add delete tracking
> ----------------------------
>
>                 Key: HADOOP-13760
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13760
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Sean Mackrory
>         Attachments: HADOOP-13760-HADOOP-13345.001.patch, HADOOP-13760-HADOOP-13345.002.patch,
HADOOP-13760-HADOOP-13345.003.patch, HADOOP-13760-HADOOP-13345.004.patch, HADOOP-13760-HADOOP-13345.005.patch,
HADOOP-13760-HADOOP-13345.006.patch, HADOOP-13760-HADOOP-13345.007.patch, HADOOP-13760-HADOOP-13345.008.patch,
HADOOP-13760-HADOOP-13345.009.patch, HADOOP-13760-HADOOP-13345.010.patch, HADOOP-13760-HADOOP-13345.011.patch,
HADOOP-13760-HADOOP-13345.012.patch, HADOOP-13760-HADOOP-13345.013.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add delete
tracking.
> Current behavior on delete is to remove the metadata from the MetadataStore.  To make
deletes consistent, we need to add a {{isDeleted}} flag to {{PathMetadata}} and check it when
returning results from functions like {{getFileStatus()}} and {{listStatus()}}.  In HADOOP-13651,
I added TODO comments in most of the places these new conditions are needed.  The work does
not look too bad.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message