spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Rőczei <roc...@gmail.com>
Subject Re: Spark standalone - reading kerberos hdfs
Date Sun, 24 Jan 2021 00:12:17 GMT
Hi Sudhir,

> On 21 Jan 2021, at 16:24, Sudhir Babu Pothineni <sbpothineni@gmail.com> wrote:
> 
> Any other insights into this issue? I tried multiple way to supply keytab to executor

> 
> Does spark standalone doesn’t support Kerberos?

Spark standalone mode does not support Kerberos authentication. Related source code:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L346


    // Kerberos is not supported in standalone mode, and keytab support is not yet available
    // in Mesos cluster mode.
    if (clusterManager != STANDALONE
        && !isMesosCluster
        && args.principal != null
        && args.keytab != null) {
      // If client mode, make sure the keytab is just a local path.
      if (deployMode == CLIENT && Utils.isLocalUri(args.keytab)) {
        args.keytab = new URI(args.keytab).getPath()
      }

If you want to test your application with Kerberos, I recommend for you local mode.

https://spark.apache.org/docs/latest/submitting-applications.html#master-urls

For example:

spark-shell --master local

and if you want to access a HDFS filesystem, then you need to add the following parameter
as well: spark.kerberos.access.hadoopFileSystems (in old Spark versions this is spark.yarn.access.hadoopFileSystems)

spark-shell --master local --conf spark.kerberos.access.hadoopFileSystems=hdfs://namenode.example.com:8020

It will create for you the necessary HDFS delegation token. Here is a very good documentation
about the delegation token handling in Spark

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/security/README.md

Best regards,

      Gabor
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message