drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alonzo Barnett <abarn...@lexmark.com>
Subject Re: Drill connect to S3 with AWS EMR role
Date Tue, 12 May 2015 16:04:16 GMT
Access to S3 from AWS emr with IAM roles seems to work fine with AMI 3.7.0.
 the core-site.xml has been modified to call an AWS java
package com.amazon.ws.emr.hadoop.fs.EmrFileSystem which I assume implements
the key rotation.  This is all done by the EMR service when the cluster is
generated.

Any clue how to let drill know that this can be called instead of looking
for fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey ? The core-site.xml
for the EMR I am using is below.


<property><name>hadoop.proxyuser.hadoop.groups</name><value>*</value></property>

<property><name>fs.s3n.impl</name><value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value></property>

<property><name>fs.s3.impl</name><value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value></property>
  <property><name>fs.default.name
</name><value>hdfs://xxx.xxx.xxx.xxx:9000</value></property>

<property><name>hadoop.tmp.dir</name><value>/mnt/var/lib/hadoop/tmp</value></property>

<property><name>fs.s3.buffer.dir</name><value>/mnt/var/lib/hadoop/s3,/mnt1/var/lib/hadoop/s3</value></property>

<property><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value></property>

<property><name>fs.s3bfs.impl</name><value>org.apache.hadoop.fs.s3.S3FileSystem</value></property>

<property><name>hadoop.metrics.defaultFile</name><value>/home/hadoop/conf/hadoopDefaultMetricsList</value></property>

<property><name>hadoop.proxyuser.hadoop.hosts</name><value>*</value></property>

<property><name>hadoop.metrics.list</name><value>TotalLoad,CapacityTotalGB,UnderReplicatedBlocks,CapacityRemainingGB,PendingDeletionBlocks,PendingReplicationBlocks,CorruptBlocks,CapacityUsedGB,numLiveDataNodes,numDeadDataNodes,MissingBlocks</value></property>

<property><name>io.compression.codec.lzo.class</name><value>com.hadoop.compression.lzo.LzoCodec</value></property>
</configuration>


On Tue, May 12, 2015 at 9:13 AM, Alonzo Barnett <abarnett@lexmark.com>
wrote:

> Thank you for the help.
>
> I am using the jetS3t support. I will look into setting up emrfs support
> today, though, and share anything I come up with.
>
>
> Sent via the Samsung Galaxy Note® 4, an AT&T 4G LTE smartphone
>
>
> -------- Original message --------
> From: David Tucker <dtucker@maprtech.com>
> Date: 05/12/2015 1:18 AM (GMT-05:00)
> To: user@drill.apache.org
> Subject: Re: Drill connect to S3 with AWS EMR role
>
> The current version of Hadoop in EMR (both Apache and MapR) does not
> support the IAM authentication to S3 without the credentials in core-site.
>  I believe the support has been integrated into Hadoop 2.6 … so when the
> EMR distributions upgrade to that level, the access you request should be
> supported.
>
> Did you successfully configure the drill-bit to use the full EMRFS jars,
> or did you default to the older jets3t support ?   If you have the
> classpath settings for full emrfs support, please share them with the group
> (and I will integrate that support into the MapR EMR bootstrap action under
> development for Drill).
>
> Regards,
>    David
>
> On May 11, 2015, at 6:55 PM, Alonzo Barnett <abarnett@lexmark.com> wrote:
>
> > I am looking into using Drill with AWS EMR.  My organization uses IAM
> roles
> > with EMR in order to rotate security credentials.
> >
> > In a first round of testing I successfully connected Drill to S3 on a
> > cluster without IAM role based credential rotation which required placing
> > access and secret keys in core-site.xml.
> >
> > It is possible to still use Drill with S3 without hardcoding credentials
> > into core-site?  With emrfs there is some work by AWS to rotate
> > credentials, and I would like to exploit the modification to core-site if
> > possible.
> >
> > --
> > v/r,
> > Alonzo Barnett
>
>


-- 
v/r,
Alonzo Barnett
Data Scientist
Desk: (859) 232 - 6848
Mobile: (571) 406 - 3391

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message