spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bbailey <>
Subject Re: S3 Bucket Access
Date Wed, 21 Jan 2015 01:15:13 GMT
Hi sranga,

Were you ever able to get authentication working with the temporary IAM
credentials (id, secret, & token)? I am in the same situation and it would
be great if we could document a solution so others can benefit from this 


sranga wrote
> Thanks Rishi. That is exactly what I am trying to do now :)
> On Tue, Oct 14, 2014 at 2:41 PM, Rishi Pidva &lt;

> rpidva@

> &gt; wrote:
>> As per EMR documentation:
>> Access AWS Resources Using IAM Roles
>> If you've launched your cluster with an IAM role, applications running on
>> the EC2 instances of that cluster can use the IAM role to obtain
>> temporary
>> account credentials to use when calling services in AWS.
>> The version of Hadoop available on AMI 2.3.0 and later has already been
>> updated to make use of IAM roles. If your application runs strictly on
>> top
>> of the Hadoop architecture, and does not directly call any service in
>> AWS,
>> it should work with IAM roles with no modification.
>> If your application calls services in AWS directly, you'll need to update
>> it to take advantage of IAM roles. This means that instead of obtaining
>> account credentials from/home/hadoop/conf/core-site.xml on the EC2
>> instances in the cluster, your application will now either use an SDK to
>> access the resources using IAM roles, or call the EC2 instance metadata
>> to
>> obtain the temporary credentials.
>> --
>> Maybe you can use AWS SDK in your application to provide AWS credentials?
>> On Oct 14, 2014, at 11:10 AM, Ranga &lt;

> sranga@

> &gt; wrote:

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message