hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
Date Mon, 12 Nov 2018 11:47:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683651#comment-16683651

Steve Loughran commented on HADOOP-14556:


good q. 

# Dtutil only fetches DTs if UGI is in secure mode, whereas fetchdt asks the FS irrespective
of the local security state. Therefore it can issue DTs without Kerberos. You can't use them
for job submission as MR's token fetching (also used by Distcp) requires Kerberos, as does
the spark token collection. But you can use the tokens collected by fetchdt in other apps,
as the [latest relase of cloudstore does|https://github.com/steveloughran/cloudstore/releases/tag/tag_2018_11_09b]

# Because the probe for "Are tokens available" doesn't take the FS URI , the impl has to say
"yes" without knowing if the FS actually does.

# Dtutil expects that when a token is requested, the impl always returns 1+ token. Because
s3a token issuing is optional (as it is on azure, abfs), if you ask the FS for a token and
it doesn't issue one, you get a stack trace (Array out of bounds or something similar)

For fetch DT to work in this world, it needs

* service loading to be resilient to classpath problems (FWIW, so does whole token mechanism:
* FS (or at least s3a FS) code to say "true" whenever probed to see if tokens are available
* dtutil to be ready to handle the case where "no tokens actually get issued" (at the very
least make it an option)

that means: changes in DTutil, and the fs binding

> S3A to support Delegation Tokens
> --------------------------------
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch,
HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch,
HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, HADOOP-14556-015.patch,
HADOOP-14556-016.patch, HADOOP-14556-017.patch, HADOOP-14556-018a.patch, HADOOP-14556.oath-002.patch,
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these
will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate
the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial
duration. Also, as you can't request an STS token from a temporary session, IAM instances
won't be able to issue tokens.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message