hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
Date Wed, 10 Jan 2018 15:46:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320470#comment-16320470

Rushabh S Shah commented on HADOOP-14445:

Attaching a new patch for trunk which fixes the incompatibility  that were discussed in previous
comments and addressing the review comments made by Xiao on the previous patch.
Added a bunch of test cases to verify compatibility.
bq. I think we can improve the comments with dtService to be clearer. Suggest something along
the lines of:
Addressed in latest patch.

bq. Now that we handle the port stuff nicely in unit tests, fallbackDefaultPortForTesting
and related logic can be removed.
Addressed in latest patch.

bq. On renew and cancel, suggest to add a debug log when keyProvider == null for supportability.
Addressed in latest patch.

bq. Let's use HADOOP_SECURITY_KEY_PROVIDER_PATH instead of KeyProviderFactory.KEY_PROVIDER_PATH.
Addressed in latest patch.

bq. When createKeyProviderForTests returns non-null value (before return kp), add a info log,
since this should only happen in tests
Addressed in latest patch.

bq. doKMSWithZKWithDelegationToken, do we need to loop through the tokens and verify? After
this fix, there would only be 1 kms-dt mapping to the entire url right? IMO we should verify
there's just 1 kms-dt.
Addressed in latest patch.

bq. doKMSWithZKWithDelegationToken, besides verifying renewal, we should also verify some
key operations.
This patch has nothing to do with key operations. Key shell commands don't use delegation
tokens. They use kerberos tickets.
This jira is only changing delegation token handling part. The existing key shell tests are
If you think existing tests are not enough, then please open a new jira to cover that.

bq. Happy to see the compat test, thanks! We should also verify some key operations too here.
Same comment as last one.

bq.HdfsKMSUtil: Looks like we can remove the not-used createKeyProvider method.
This method is getting called from {{DFSUtil#createKeyProviderCryptoExtension}}. Namenode
calls this method to create key provider.
So we cannot remove this method.
[~xiaochen]: please review.

> Delegation tokens are not shared between KMS instances
> ------------------------------------------------------
>                 Key: HADOOP-14445
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14445
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: kms
>    Affects Versions: 2.8.0, 3.0.0-alpha1
>         Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>            Reporter: Wei-Chiu Chuang
>            Assignee: Rushabh S Shah
>         Attachments: HADOOP-14445-branch-2.8.002.patch, HADOOP-14445-branch-2.8.patch,
HADOOP-14445.002.patch, HADOOP-14445.003.patch
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do not share
delegation tokens. (a client uses KMS address/port as the key for delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
>         InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
>             url.getPort());
>         Text service = SecurityUtil.buildTokenService(serviceAddr);
>         dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation tokens
> Under HA, A KMS instance must verify the delegation token given by another KMS instance,
by checking the shared secret used to sign the delegation token. To do this, all KMS instances
must be able to retrieve the shared secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share delegation tokens.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message