spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eabour@163.com" <eab...@163.com>
Subject Is there a better way to read kerberized impala tables by spark jdbc?
Date Tue, 08 Dec 2020 02:54:35 GMT
Hi:

I want to use spark jdbc to read kerberized impala tables, like:
```
val impalaUrl = "jdbc:impala://<host_imapal_deamon>:21050;AuthMech=1;KrbRealm=REALM.COM;KrbHostFQDN=<host_impala_deamon>;KrbServiceName=impala"
spark.read.jdbc(impalaUrl)
```

As we know, spark will read impala data by executor rather than driver, so throw excepting:
 javax.security.sasl.SaslException: GSS initiate failed

```
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        ... 20 common frames omitted

``` 

Ony way to solve this problem is set jaas.conf by "java.security.auth.login.config" property,


This is jaas.conf:

```
Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      doNotPrompt=true
      useTicketCache=true
      principal="test"
      keyTab="/home/keytab/user.keytab";
   };

```

Then set spark.executor.extraJavaOptions like :
```
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/data/disk1/spark-jdbc-impala/conf/jaas.conf
-Djavax.security.auth.useSubjectCredsOnly=false" 
```

This way required absolute jaas.conf file and keyTab file, in other words, these files must
be placed in the same path and on each node, Is there a better way?

Please help.

Regards




eabour@163.com
Mime
View raw message