hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Budde (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11670) Fix IAM instance profile auth for s3a
Date Fri, 06 Mar 2015 21:20:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350903#comment-14350903

Adam Budde commented on HADOOP-11670:

What is the proper way to execute the hadoop-aws tests? I'm trying to execute 'mvn test -Ptests-on'
in the hadoop-tools/hadoop-aws dir and every tests fails with the following exception:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed
in 8.0
Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.239 sec <<< FAILURE!
- in org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
testBasicReadWriteIO(org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem)  Time elapsed:
0.209 sec  <<< ERROR!
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/mnt/md0/build/hadoop/hadoop-tools/hadoop-aws/target/test-classes/core-site.xml;
lineNumber: 47; columnNumber: 36; An include with href 'auth-keys.xml'failed, and no fallback
element was found.
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:348)
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
        at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2499)
        at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2487)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2558)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2511)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2424)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:998)
        at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1048)
        at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1449)
        at org.apache.hadoop.fs.FileSystem.initialize(FileSystem.java:204)
        at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:322)
        at org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem.setUp(TestS3NInMemoryFileSystem.java:44)

> Fix IAM instance profile auth for s3a
> -------------------------------------
>                 Key: HADOOP-11670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11670
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.7.0
>            Reporter: Adam Budde
>            Assignee: Adam Budde
>             Fix For: 2.7.0
>         Attachments: HADOOP-11670-001.patch, HADOOP-11670.002.patch
> One big advantage provided by the s3a filesystem is the ability to use an IAM instance
profile in order to authenticate when attempting to access an S3 bucket from an EC2 instance.
This eliminates the need to deploy AWS account credentials to the instance or to provide them
to Hadoop via the fs.s3a.awsAccessKeyId and fs.s3a.awsSecretAccessKey params.
> The patch submitted to resolve HADOOP-10714 breaks this behavior by using the S3Credentials
class to read the value of these two params. The change in question is presented below:
> S3AFileSystem.java, lines 161-170:
> {code}
>     // Try to get our credentials or just connect anonymously
>     S3Credentials s3Credentials = new S3Credentials();
>     s3Credentials.initialize(name, conf);
>     AWSCredentialsProviderChain credentials = new AWSCredentialsProviderChain(
>         new BasicAWSCredentialsProvider(s3Credentials.getAccessKey(),
>                                         s3Credentials.getSecretAccessKey()),
>         new InstanceProfileCredentialsProvider(),
>         new AnonymousAWSCredentialsProvider()
>     );
> {code}
> As you can see, the getAccessKey() and getSecretAccessKey() methods from the S3Credentials
class are now used to provide constructor arguments to BasicAWSCredentialsProvider. These
methods will raise an exception if the fs.s3a.awsAccessKeyId or fs.s3a.awsSecretAccessKey
params are missing, respectively. If a user is relying on an IAM instance profile to authenticate
to an S3 bucket and therefore doesn't supply values for these params, they will receive an
exception and won't be able to access the bucket.

This message was sent by Atlassian JIRA

View raw message