hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabor Bota (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-16280) S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened
Date Tue, 30 Apr 2019 12:50:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gabor Bota updated HADOOP-16280:
--------------------------------
    Description: 
When using S3Guard in authoritative mode a file can be reported from AWS S3 that's missing
like it is described in the following exception:
{noformat}
java.io.FileNotFoundException: re-open s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
at 0 on s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service:
Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended
Request ID: NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
S3 Extended Request ID: NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
{noformat}

But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened. The operation will
not fail when it's opened, it will fail when we try to read it, so the call
{noformat}FSDataInputStream is = guardedFs.open(testFilePath);{noformat}
won't fail, but the next call
{noformat}
byte[] firstRead = new byte[text.length()];
is.read(firstRead, 0, firstRead.length);
{noformat}
will fail with the exception message like what's above.


Once Authoritative mode is on, we assume that there is no out of band operation, so the file
will appear eventually. We should re-try in this case.

  was:
When using S3Guard in authoritative mode a file can be reported from AWS S3 that's missing
like it is described in the following exception:
{noformat}
java.io.FileNotFoundException: re-open s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
at 0 on s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service:
Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended
Request ID: NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
S3 Extended Request ID: NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
{noformat}

But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened. The operation will
not fail when it's opened, it will fail when we try to read it, so the call
{noformat}FSDataInputStream is = guardedFs.open(testFilePath);{noformat}
won't fail, but the next call
{noformat}      
byte[] firstRead = new byte[text.length()];
is.read(firstRead, 0, firstRead.length);
{noformat}
will fail with the exception message like what's above.


Once Authoritative mode is on, we assume that there is no out of band operation, so the file
will appear eventually. We should re-try in this case.


> S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-16280
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16280
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Gabor Bota
>            Priority: Major
>
> When using S3Guard in authoritative mode a file can be reported from AWS S3 that's missing
like it is described in the following exception:
> {noformat}
> java.io.FileNotFoundException: re-open s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
at 0 on s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service:
Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended
Request ID: NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
S3 Extended Request ID: NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
> {noformat}
> But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened. The operation
will not fail when it's opened, it will fail when we try to read it, so the call
> {noformat}FSDataInputStream is = guardedFs.open(testFilePath);{noformat}
> won't fail, but the next call
> {noformat}
> byte[] firstRead = new byte[text.length()];
> is.read(firstRead, 0, firstRead.length);
> {noformat}
> will fail with the exception message like what's above.
> Once Authoritative mode is on, we assume that there is no out of band operation, so the
file will appear eventually. We should re-try in this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message