spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Foster Langbein <>
Subject kerberos auth for MS SQL server jdbc driver
Date Mon, 15 Oct 2018 07:03:15 GMT
Has anyone gotten spark to write to SQL server using Kerberos
authentication with Microsoft's JDBC driver? I'm having limited success,
though in theory it should work.

I'm using a YARN-mode 4-node Spark 2.3.0 cluster and trying to write a
simple table to SQL Server 2016. I can get it to work if I use SQL server
credentials, however this is not an option in my application. I need to use
windows authentication - so-called integratedSecurity - and in particular I
want to use a keytab file.

The solution half works - the spark driver creates a table on SQL server -
so I'm pretty confident the Kerberos implementation/credentials etc are
setup correctly and valid. However the executors then fail to write any
data to the table with an exception: "
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)"

After much tracing/debugging it seems executors are behaving differently to
the spark driver and ignoring the specification to use the credentials
supplied in the keytab and instead trying to use the default spark cluster
user. I simply haven't been able to force them to use what's in the keytab
after trying many. many variations.

Very grateful if anyone has any help/suggestions/ideas on how to get this
to work.


*Dr Foster Langbein* | Chief Technology Officer | Risk Frontiers

Level 2, 100 Christie St, St Leonards, NSW, 2065

Telephone: +61 2 8459 9777

Email: | Website:

*Risk Modelling | Risk Management | Resilience | Disaster Management
| Social Research Australia | New Zealand | Asia Pacific*

View raw message