hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
Date Fri, 01 Apr 2016 12:47:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221648#comment-15221648

Steve Loughran commented on HADOOP-12563:

Here are the use cases I've encountered related to this

* saving tokens for a principal to a file (HDFS, RM, ATS), so that a process can be started
in an env with {{HADOOP_TOKEN_FILE_LOCATION}} pointing at the file. This lets me test oozie
deployment outside of oozie.
* spark yarn client having to pick up tokens for HBase, Hive and others. This is done on a
case-by-case basis through introspection ugliness. With a standard interface, all you'd need
to do is load the implementation and invoke.
* spark AM doing ticket-based token retrieval, for propagation to executors in containers.

So: one similar, two within an app, all benefiting from a standard API. Use case #1 can be
handled by your CLI tool, if it does keytab and principal

1. return values: There was a comment about returning null; I want to make sure that that
is not the case, the failures -> exceptions
2. the rationale for {{isTokenRequired()}} is related to other uses. Example, spark only needs
an hbase token if (a) hbase is on the classpath, (b) the hbase-site.xml provides the binding
for hbase and indicates that authentication is needed. You may have unauthed hbase within
a kerberized cluster. Similarly for RM web access, the implementation would look at the auth
method for the web UI; again, there may be none, even on a secure cluster.

3. RM And ATS code can be found in these classes


4. keytabs & principals

bq. I agree it would be cool to have some mechanism to let hadoop know how to kinit for an
OS user who is already authenticated and has OS perms to access a keytab, e.g. "kinit -kt

it's called {{UserGroupInformation.loginUserFromKeytabAndReturnUGI()}}, and is easy to use,
provided you make it the first thing you do in your code after reading all config, and before
talking to any services. Look in {{TokensOperation}} for the code to lift

Finally, note that token acquisition on HA clusters is tricker than you' expect ... we'll
all need to review that code.

> Updated utility to create/modify token files
> --------------------------------------------
>                 Key: HADOOP-12563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12563
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: Matthew Paduano
>         Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, HADOOP-12563.03.patch,
HADOOP-12563.04.patch, HADOOP-12563.05.patch, HADOOP-12563.06.patch, HADOOP-12563.07.patch,
HADOOP-12563.07.patch, HADOOP-12563.08.patch, dtutil-test-out, dtutil_diff_07_08, example_dtutil_commands_and_output.txt,
> hdfs fetchdt is missing some critical features and is geared almost exclusively towards
HDFS operations.  Additionally, the token files that are created use Java serializations which
are hard/impossible to deal with in other languages. It should be replaced with a better utility
in common that can read/write protobuf-based token files, has enough flexibility to be used
with other services, and offers key functionality such as append and rename. The old version
file format should still be supported for backward compatibility, but will be effectively
> A follow-on JIRA will deprecrate fetchdt.

This message was sent by Atlassian JIRA

View raw message