spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Rőczei <>
Subject Re: Spark standalone - reading kerberos hdfs
Date Sun, 24 Jan 2021 00:12:17 GMT
Hi Sudhir,

> On 21 Jan 2021, at 16:24, Sudhir Babu Pothineni <> wrote:
> Any other insights into this issue? I tried multiple way to supply keytab to executor

> Does spark standalone doesn’t support Kerberos?

Spark standalone mode does not support Kerberos authentication. Related source code:

    // Kerberos is not supported in standalone mode, and keytab support is not yet available
    // in Mesos cluster mode.
    if (clusterManager != STANDALONE
        && !isMesosCluster
        && args.principal != null
        && args.keytab != null) {
      // If client mode, make sure the keytab is just a local path.
      if (deployMode == CLIENT && Utils.isLocalUri(args.keytab)) {
        args.keytab = new URI(args.keytab).getPath()

If you want to test your application with Kerberos, I recommend for you local mode.

For example:

spark-shell --master local

and if you want to access a HDFS filesystem, then you need to add the following parameter
as well: spark.kerberos.access.hadoopFileSystems (in old Spark versions this is spark.yarn.access.hadoopFileSystems)

spark-shell --master local --conf spark.kerberos.access.hadoopFileSystems=hdfs://

It will create for you the necessary HDFS delegation token. Here is a very good documentation
about the delegation token handling in Spark

Best regards,

To unsubscribe e-mail:

View raw message