spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (Jira)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials
Date Fri, 01 May 2020 16:52:02 GMT

     [ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-31551:
------------------------------------

    Assignee: Apache Spark

> createSparkUser lost user's non-Hadoop credentials
> --------------------------------------------------
>
>                 Key: SPARK-31551
>                 URL: https://issues.apache.org/jira/browse/SPARK-31551
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.4, 2.4.5
>            Reporter: Yuqi Wang
>            Assignee: Apache Spark
>            Priority: Major
>
> See current *[createSparkUser|https://github.com/apache/spark/blob/263f04db865920d9c10251517b00a1b477b58ff1/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L66-L76]*:
> {code:java}
>    def createSparkUser(): UserGroupInformation = {
>     val user = Utils.getCurrentUserName()
>     logDebug("creating UGI for user: " + user)
>     val ugi = UserGroupInformation.createRemoteUser(user)
>     transferCredentials(UserGroupInformation.getCurrentUser(), ugi)
>     ugi
>   }
>   def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation):
Unit = {
>     dest.addCredentials(source.getCredentials())
>   }
>   def getCurrentUserName(): String = {
>     Option(System.getenv("SPARK_USER"))
>       .getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
>   }
> {code}
> The *transferCredentials* func can only transfer Hadoop creds such as Delegation Tokens.
>  However, other creds stored in UGI.subject.getPrivateCredentials, will be lost here,
such as:
>  # Non-Hadoop creds:
>  Such as, [Kafka creds |https://github.com/apache/kafka/blob/f3c8bff311b0e4c4d0e316ac949fe4491f9b107f/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginModule.java#L395]
>  # Newly supported or 3rd party supported Hadoop creds:
>  Such as to support OAuth/JWT token authn on Hadoop, we need to store the OAuth/JWT token
into UGI.subject.getPrivateCredentials. However, these tokens are not supposed to be managed
by Hadoop Credentials (currently it is only for Hadoop secret keys and delegation tokens)
> Another issue is that the *SPARK_USER* only gets the UserGroupInformation.getCurrentUser().getShortUserName()
of the user, which may lost the user's fully qualified user name. We should better use the
*getUserName* to get fully qualified user name in our client side, which is aligned to *[HADOOP_PROXY_USER|https://github.com/apache/hadoop/blob/30ef8d0f1a1463931fe581a46c739dad4c8260e4/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L716-L720]*.
> Related to https://issues.apache.org/jira/browse/SPARK-1051



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message