spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mridul Muralidharan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
Date Tue, 10 Jan 2017 20:08:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816026#comment-15816026
] 

Mridul Muralidharan commented on SPARK-19143:
---------------------------------------------



bq. You say "we added", you are saying you have already implemented this?

Yes, though it is not for master - we are still ironing out issues as they arise : the actual
mechanics of how the update is propagated is left as implementation, in our case, we leveraged
livy rpc to push updates which are then applied on the remote session.

The impl leverages existing hdfs based distribution to minimize change - but as [~vanzin]
mentioned, other options can be explored ofcourse, though they might be more intrusive (broadcast
approach, for example, will mean some way to trigger a task or force each 'Task' execution
to update tokens if available : needs more thought, atleast from me !)

> API in Spark for distributing new delegation tokens (Improve delegation token handling
in secure clusters)
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19143
>                 URL: https://issues.apache.org/jira/browse/SPARK-19143
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 2.0.2, 2.1.0
>            Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments| https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens. Quoting [~vanzin]
from SPARK-5493 
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the TTL, the
session is unusable.
> There might be ways to hack support for that without changes in Spark, but I'd like to
see a proper API in Spark for distributing new delegation tokens. I mentioned that in SPARK-14743,
but although that bug is closed, that particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message