spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Help accessing protected S3
Date Thu, 23 Jul 2015 17:36:28 GMT

> On 23 Jul 2015, at 01:50, Ewan Leith <ewan.leith@realitymine.com> wrote:
> 
> I think the standard S3 driver used in Spark from the Hadoop project (S3n) doesn't support
IAM role based authentication.
> 
> However, S3a should support it. If you're running Hadoop 2.6 via the spark-ec2 scripts
(I'm not sure what it launches with by default) try accessing your bucket via s3a:// URLs
instead of s3n://
> 
> http://wiki.apache.org/hadoop/AmazonS3
> 
> https://issues.apache.org/jira/browse/HADOOP-10400
> 
> Thanks,
> Ewan
> 

s3a should support roles. note that it isn't ready for production use before Hadoop 2.7.1,
various scaiability and performance problems surfaced after 2.6 shipped

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message