airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Crawford <>
Subject Re: hiding aws secret key in connections
Date Wed, 20 Sep 2017 22:47:19 GMT
After some more research it appears that aws_hook in contrib/hooks actually does do it the
way I proposed with looking at the login and password of the connection, but it doesn’t
look at the extra json for the access and secret keys.

def get_client_type(self, client_type, region_name=None):
        connection_object = self.get_connection(self.aws_conn_id)
        aws_access_key_id = connection_object.login
        aws_secret_access_key = connection_object.password

        if region_name is None:
            region_name = connection_object.extra_dejson.get('region_name')

    except AirflowException:
        # No connection found: fallback on boto3 credential strategy
        aws_access_key_id = None
        aws_secret_access_key = None

    return boto3.client(
However the S3Hook looks for this info in a different using the older boto library instead
of boto3.  

So it appears we have 2 different parts of airflow interacting with aws but specifying their
credentials in different ways.


> On Sep 19, 2017, at 12:01 PM, Ali Uz <> wrote:
> We use a dynamic config where we iterate through a JSON file, and all
> sensitive info (like api keys, aws keys, etc...) are pulled from a remote
> k/v store when airflow starts and adds them as fields to the JSON config
> file.
> On Tue, Sep 19, 2017 at 6:54 PM, Michael Crawford <
>> wrote:
>> Did my message go through?  I have never tried to send an email to the
>> list before, only silently monitored.
>> Does anyone have any ideas?   I would be happy to create an issue and code
>> up the fix myself, but I just wanted to ping here first to make sure I
>> wasn’t missing anything and try to get a consensus on how to handle this.
>> Thanks,
>> Mike
>>> On Sep 18, 2017, at 8:03 PM, Michael Crawford <michael.crawford@
>>> wrote:
>>> Hi,
>>> I was wondering if anything had ever been proposed for having the aws
>> secret key hidden in the aws type connection.
>>> Currently passing in these credentials is done by defining the some json
>> in the extra params section of the connection like
>>> {"aws_access_key_id":"_your_aws_access_key_id_",
>> "aws_secret_access_key": "_your_aws_secret_access_key_”}
>>> While this does work it leaves the secret access key in plain text for
>> anyone that has access to the connections.
>>> I know there are other options about setting them as environment
>> variables, but this doesn’t help if we need to define more than one aws
>> connection with different access keys.
>>> Two things that immediately came to mind for how to do this:
>>> 1.  use login and password sections of the connection for the access and
>> secret keys so that the secret gets hidden and encrypted like all the other
>> passwords.
>>> 2. have an option to encrypt the extra params
>>> Option 1 seems most logical and should be too hard to implement.
>>> Open to any ideas people might have on this.
>>> Thanks,
>>> Mike

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message