hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Roling (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-16085) S3Guard: use object version to protect against inconsistent read after replace/overwrite
Date Fri, 01 Feb 2019 16:36:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758479#comment-16758479
] 

Ben Roling edited comment on HADOOP-16085 at 2/1/19 4:35 PM:
-------------------------------------------------------------

I've attached my first version of the patch.  It's based on 3.2.0 rather than trunk.  I
was having a bit of trouble with the build and running the tests when I was working off master
and I switched over to 3.2.0.  I need to port it back over to trunk but there are some conflicts
I need to resolve in doing so.  Anyway, if anyone wants to have a look at the initial patch
it should give you a general idea of the changes.  It's not terribly complicated.

Bear in mind, I am still also evaluating switching over to etags instead of versionId.


was (Author: ben.roling):
I've attached my first version of the patch.  It's based on 3.2.0 rather than trunk.  I
was having a bit of trouble with the build and running the tests when I was working off master
and I switched over to 3.2.0.  I need to port it back over to trunk but there are some conflicts
I need to resolve in doing so.  Anyway, if anyone wants to have a look at the initial patch
it should give you a general idea of the changes.  It's not terribly complicated.

> S3Guard: use object version to protect against inconsistent read after replace/overwrite
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-16085
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16085
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Ben Roling
>            Priority: Major
>         Attachments: HADOOP-16085_3.2.0_001.patch
>
>
> Currently S3Guard doesn't track S3 object versions.  If a file is written in S3A with
S3Guard and then subsequently overwritten, there is no protection against the next reader
seeing the old version of the file instead of the new one.
> It seems like the S3Guard metadata could track the S3 object version.  When a file is
created or updated, the object version could be written to the S3Guard metadata.  When a
file is read, the read out of S3 could be performed by object version, ensuring the correct
version is retrieved.
> I don't have a lot of direct experience with this yet, but this is my impression from
looking through the code.  My organization is looking to shift some datasets stored in HDFS
over to S3 and is concerned about this potential issue as there are some cases in our codebase
that would do an overwrite.
> I imagine this idea may have been considered before but I couldn't quite track down any
JIRAs discussing it.  If there is one, feel free to close this with a reference to it.
> Am I understanding things correctly?  Is this idea feasible?  Any feedback that could
be provided would be appreciated.  We may consider crafting a patch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message