spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Masiero Vanzin (Jira)" <>
Subject [jira] [Resolved] (SPARK-31559) AM starts with initial fetched tokens in any attempt
Date Tue, 12 May 2020 00:27:00 GMT


Marcelo Masiero Vanzin resolved SPARK-31559.
    Fix Version/s: 3.0.0
         Assignee: Jungtaek Lim
       Resolution: Fixed

> AM starts with initial fetched tokens in any attempt
> ----------------------------------------------------
>                 Key: SPARK-31559
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>             Fix For: 3.0.0
> The issue is only occurred in yarn-cluster mode.
> Submitter will obtain delegation tokens for yarn-cluster mode, and add these credentials
to the launch context. AM will be launched with these credentials, and AM and driver are able
to leverage these tokens.
> In Yarn cluster mode, driver is launched in AM, which in turn initializes token manager
(while initializing SparkContext) and obtain delegation tokens (+ schedule to renew) if both
principal and keytab are available.
> That said, even we provide principal and keytab to run application with yarn-cluster
mode, AM always starts with initial tokens from launch context until token manager runs and
obtains delegation tokens.
> So there's a "gap", and if user codes (driver) access to external system with delegation
tokens (e.g. HDFS) before initializing SparkContext, it cannot leverage the tokens token manager
will obtain. It will make the application fail if AM is killed "after" the initial tokens
are expired and relaunched.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message