hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
Date Tue, 27 Nov 2018 16:22:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700653#comment-16700653

Steve Loughran commented on HADOOP-14556:

Hi [~elgoiri]: thanks for this review; not had a chance to reply until now.

bq. The unit tests cover the basic cases well.

I'd have liked to have a real mini-yarn cluster with distcp, but couldn't get kerberos to
work with miniyarn and minihdfs to the extent the cluster would come up. If/when someone can
do that. I'd revisit it.

bq. Very long patch and even though there are a bunch of interfaces which are pretty verbose,
there is a lot here. I'm not sure if there are ways to split it. For example the utilities
to fetch the DT.

I know, and I always worry about adding more complexity for the following reason: other people
have to maintain it, and if they can't either the code is neglected or I'm expected to be
the maintainer indefinitely. 

I've tried to keep all DT support out in its own home, with not that much in the S3A FS -but
as I changed the encryption stuff there may be too much of a diff there. I could perhaps revert
some of that. Less elegant but a smaller diff for that file, and so less risk of merge conflict.

And because I was going near session credential management, I also tried to coalesce stuff
that the credential providers were doing. Again, I could look to pull that for now

Otherwise: I've needed to do all 3 including the role stuff, to make sure I hadn't blocked
out those. I even believe that I've done enough to support more advanced bindings. We could
strip out the full credentials as it doesn't reduce risk, and so only support session and
role secrets? that'd work well for locking down AWS, but I would also like to support third
party stores which don't have sessions

regarding the docs, [~lmccay] has suggested I could actually do a video of this at work. Would
people be interested? That'd be a real demo of role-base-DT => live cluster for distcp.

> S3A to support Delegation Tokens
> --------------------------------
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch,
HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch,
HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, HADOOP-14556-015.patch,
HADOOP-14556-016.patch, HADOOP-14556-017.patch, HADOOP-14556-018a.patch, HADOOP-14556-019.patch,
HADOOP-14556-020.patch, HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these
will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate
the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial
duration. Also, as you can't request an STS token from a temporary session, IAM instances
won't be able to issue tokens.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message