Hello all,
I apologize in advance if this seems like a silly question, however I've
not been able to determine if Scoop supports Kerberos delegation on
import, down to the MapReduce process.
I have a user who has a valid Kerberos ticket that supports forwarding,
additionally they are working on a completely kerberized Hadoop cluster
and all hosts in the cluster are trusted for kerberos delegation.
We have been able to use Scoop with Kerberos Authentication when
connecting to a Microsoft SQL Server using the following command:
sqoop
list-databases --connect
"jdbc:sqlserver://hostname.domain.com;integratedSecurity=true;authenticationScheme=JavaKerberos"
--username 'DOMAIN\USER'
So we tried to do an import using Kerberos, but it has failed with an
error that indicates that MapReduce does not have a kerberos ticket.
Here is the command used on the Sqoop import.
sqoop import --connect
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos;database=database_name"
--username 'Domain\user' --query 'select * from table where $CONDITIONS'
--split-by Abc_Number --target-dir /user/user/Sql
Here is the error:
INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service:
ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO
impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to
track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job:
Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce
0% INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0,
Status : FAILED Error: java.lang.RuntimeException:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication
failed. ClientConnectionId:366c38c7-c at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication
failed. ClientConnectionId:366c38c7-c at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException:
Integrated authentication failed. ClientConnectionId:366c38c7-c at
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
at
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
at java.sql.DriverManager.getConnection(DriverManager.java:664) at
java.sql.DriverManager.getConnection(DriverManager.java:247) at
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more Caused by: java.security.PrivilegedActionException:
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150)
... 26 more Caused by: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt) at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at
sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at
sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187)
... 30 more
Does anyone know if Scoop support forwarding the Kerberos ticket? and if
so, what do we need to configure to have it do so? Any comments on how
this should work, or websites discussing how you got this working in your
environment?
There is a Hortonworks Community post on this as well:
https://community.hortonworks.com/questions/52137/sqlserver-sqoop-import-integrated-security-kerbero.html
I would like to verify each step of the process can support Kerberos, and
then verify the configurations to determine if there is any configuration
set that might prevent this from working.
Any help would be appreciated.
thanks.
Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
(414-357-4062) | jkaufman@footlocker.com
Visit us on-line at footlocker.com.
The information in this e-mail, and any attachment therein, is
confidential and for use by the addressee only. If you are not the
intended recipient, please return the e-mail to the sender and delete it
from your computer. Although the Company attempts to sweep e-mail and
attachments for viruses, it does not guarantee that either are virus-free
and accepts no liability for any damage sustained as a result of viruses.
|