hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
Date Fri, 29 Jan 2016 21:31:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124252#comment-15124252
] 

Steve Loughran commented on HADOOP-12563:
-----------------------------------------

h2. list

when listing tokens
  # attempt to unmarshall them, then call toString() in the value
  # and if they are delegation tokens, print out the expiry time in a human form.
  
Add a command to verify that a token for a service exists, and is currently valid.

h2. DtFetcher

I really like the idea of a standard fetch interface, which is of tangible benefit to any
application that wants to be able to load tokens off remote services, without having to compile
in support for that service. I would point to fun Spark has to go through to get HBase and
Hive tokens as an example, as well as observing that to add support for a new service (e.g.
Kafka) will require some more reflection pain (and/or spark implements its own api).

Accordingly, a use case that the underlying code must support here is: be something usable
inside YARN applications itself.

h3. Command Shell

if we're going to do some new entry point stuff, can that be done separately? 

In particular, the YARN-679 service launcher proposes modifying ExitUtils.ExitException to
support exit codes better, and allow any raised exception to also supply an exit code. This
allows apps to fail with more meaningful errors than just "-1". We can pull that little bit
out into its own HADOOP-9626

BTW, the exit code for a missing file should be 44, which is 404 compressed into one byte.
Hence my proposed [list of exit codes|https://github.com/steveloughran/hadoop-trunk/blob/stevel/YARN-679-launcher/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/LauncherExitCodes.java]



> Updated utility to create/modify token files
> --------------------------------------------
>
>                 Key: HADOOP-12563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12563
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: Matthew Paduano
>         Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, HADOOP-12563.03.patch,
HADOOP-12563.04.patch, HADOOP-12563.05.patch, HADOOP-12563.06.patch, example_dtutil_commands_and_output.txt,
generalized_token_case.pdf
>
>
> hdfs fetchdt is missing some critical features and is geared almost exclusively towards
HDFS operations.  Additionally, the token files that are created use Java serializations which
are hard/impossible to deal with in other languages. It should be replaced with a better utility
in common that can read/write protobuf-based token files, has enough flexibility to be used
with other services, and offers key functionality such as append and rename. The old version
file format should still be supported for backward compatibility, but will be effectively
deprecated.
> A follow-on JIRA will deprecrate fetchdt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message