hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13674) S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.
Date Mon, 03 Oct 2016 20:46:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543380#comment-15543380
] 

Chris Nauroth commented on HADOOP-13674:
----------------------------------------

Here is the error I see when I attempt to access a bucket in us-west-2, but with {{fs.s3a.endpoint}}
pointing to us-west-1.

{code}
> hadoop fs -D fs.s3a.endpoint=s3-us-west-1.amazonaws.com -ls s3a://cnauroth-test-aws-s3a-logs/
ls: getFileStatus on : com.amazonaws.services.s3.model.AmazonS3Exception: The bucket you are
attempting to access must be addressed using the specified endpoint. Please send all future
requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect;
Request ID: EC6C7FCF8B40B27C), S3 Extended Request ID: EQ1h4MW2CRLV4ZJBGs2xz2CVXwsfGS5X+ByWfyl1tdzbXbf7bFn5DI5pejcWWCmu1/P/uDEOjaU=:
The bucket you are attempting to access must be addressed using the specified endpoint. Please
send all future requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code:
PermanentRedirect; Request ID: EC6C7FCF8B40B27C)
{code}

It says that the endpoint is wrong, but it doesn't say which endpoint is correct.  Turning
on debug logging shows that the information does come back in the HTTP 301 response:

{code}
> hadoop --loglevel DEBUG fs -D fs.s3a.endpoint=s3-us-west-1.amazonaws.com -ls s3a://cnauroth-test-aws-s3a-logs/
...
16/10/03 13:35:28 DEBUG http.wire:  << "<Error><Code>PermanentRedirect</Code><Message>The
bucket you are attempting to access must be addressed using the specified endpoint. Please
send all future requests to this endpoint.</Message><Bucket>cnauroth-test-aws-s3a-logs</Bucket><Endpoint>cnauroth-test-aws-s3a-logs.s3-us-west-2.amazonaws.com</Endpoint><RequestId>995927D9C5DD8F90</RequestId><HostId>LK/kvbR/gdnxyr5JXj1L41TOfcO4VBF6MtT8FkwOXXyRdjhasccrHc2bux+b4uHSqJmiBEgHJcI=</HostId></Error>"
...
{code}

It appears that the AWS SDK maps the {{<Endpoint>}} element into the map returned by
[{{AmazonS3Exception#getAdditionalDetails()}}|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/AmazonS3Exception.html#getAdditionalDetails--].
 We can use that to get the information and put it into the exception thrown from S3A.

> S3A can provide a more detailed error message when accessing a bucket through an incorrect
S3 endpoint.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13674
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>            Priority: Minor
>
> When accessing the S3 service through a region-specific endpoint, the bucket must be
located in that region.  If the client attempts to access a bucket that is not located in
that region, then the service replies with a 301 redirect and the correct region endpoint.
 However, the exception thrown by S3A does not include the correct endpoint.  If we included
that information in the exception, it would make it easier for users to diagnose and fix incorrect
configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message