hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Moist (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
Date Fri, 26 Jan 2018 23:16:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341776#comment-16341776
] 

Steve Moist commented on HADOOP-15006:
--------------------------------------

Ok fixed it.  The workflow with block output was causing it to write to disk encrypted and
then when it sent it to S3 it encrypted it again causing it to decrypt.  So there's a small
issue with that in some cases.  However, now encryption should work fine for most things. 
It uses a fixed IV and key to do the encryption, so any files written to S3 will be automatically
encrypted/decrypted, so we get some free coverage from the unit tests.  It's a quick and
dirty prototype so many of the unit tests fail as its not covering all scenarios.  I'm able
to upload/download files to S3 using the command line without issue.  When I view the object
in S3 gui, it shows up encrypted, but will automatically decrypt when i do a hdfs get from
the cli.  Play around with it and let me know what you think.  The CryptoStreams work fine,
but the integration to fully flesh this out into a feature is what we need to really look
at.

> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---------------------------------------------------------------
>
>                 Key: HADOOP-15006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15006
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3, kms
>            Reporter: Steve Moist
>            Priority: Minor
>         Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a way that
it can leverage HDFS transparent encryption, use the Hadoop KMS to manage keys, use the `hdfs
crypto` command line tools to manage encryption zones in the cloud, and enable distcp to copy
from HDFS to S3 (and vice-versa) with data still encrypted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message