spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
Date Tue, 10 Jan 2017 14:21:58 GMT


Thomas Graves commented on SPARK-19143:

[~mridulm80]  You say "we added", you are saying you have already implemented this?  

It would definitely be nice to support pushing tokens from say a gateway so you don't have
to ship the keytab because shipping the keytab is much less secure.  It would also be nice
to not use HDFS to store and transfer the tokens.  My initial thought was to create an rpc
between the client on gateways and the driver/AM (running on yarn node) and transfer the new
tokens that way.  ideally it would also then be transferred to the executors via rpc vs again
storing in hdfs.   All that would be more secure then storing in hdfs.  

We could add a command to spark-submit to get and push new credentials.  For long running
jobs it would have to happen periodically (< every 24 hours) but for initial that could
be done via cron or other mechanism.

> API in Spark for distributing new delegation tokens (Improve delegation token handling
in secure clusters)
> ----------------------------------------------------------------------------------------------------------
>                 Key: SPARK-19143
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 2.0.2, 2.1.0
>            Reporter: Ruslan Dautkhanov
> Spin off from SPARK-14743 and comments chain in [recent comments|]
in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens. Quoting [~vanzin]
from SPARK-5493 
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the TTL, the
session is unusable.
> There might be ways to hack support for that without changes in Spark, but I'd like to
see a proper API in Spark for distributing new delegation tokens. I mentioned that in SPARK-14743,
but although that bug is closed, that particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message