hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13230) S3A to optionally retain directory markers; look under a marker for files when needEmptyDir=true
Date Tue, 10 Mar 2020 22:25:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17056471#comment-17056471
] 

Aaron Fabbri commented on HADOOP-13230:
---------------------------------------

[~stevel@apache.org] I'll take a look if you want. You can always Download As -> PDF and
post that here.

> S3A to optionally retain directory markers; look under a marker for files when needEmptyDir=true
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13230
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13230
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Aaron Fabbri
>            Priority: Major
>
> Users of s3a may not realize that, in some cases, it does not interoperate well with
other s3 tools, such as the AWS CLI.  (See HIVE-13778, IMPALA-3558).
> Specifically, if a user:
> - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
> - Copies data into that directory via another tool, i.e. aws cli.
> - Tries to access the data in that directory with any Hadoop software.
> Then the last step fails because the fake empty directory blob that s3a wrote in the
first step, causes s3a (listStatus() etc.) to continue to treat that directory as empty, even
though the second step was supposed to populate the directory with data.
> I wanted to document this fact for users. We may mark this as not-fix, "by design"..
May also be interesting to brainstorm solutions and/or a config option to change the behavior
if folks care.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message