sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Kaufman <jkauf...@footlocker.com>
Subject Help using Scoop 1.4.6 with Kerberos Authentication on Import with MapReduce.
Date Wed, 07 Sep 2016 15:47:30 GMT
Hello all,

I apologize in advance if this seems like a silly question, however I've 
not been able to determine if Scoop supports  Kerberos delegation on 
import, down to the MapReduce process.

I have a user who has a valid Kerberos ticket that supports forwarding, 
additionally they are working on a completely kerberized Hadoop cluster 
and all hosts in the cluster are trusted for kerberos delegation.

We have been able to use Scoop with Kerberos Authentication when 
connecting to a Microsoft SQL Server using the following command:

sqoop
list-databases --connect
"jdbc:sqlserver://hostname.domain.com;integratedSecurity=true;authenticationScheme=JavaKerberos"
--username 'DOMAIN\USER'

So we tried to do an import using Kerberos, but it has failed with an 
error that indicates that MapReduce does not have a kerberos ticket. 

Here is the command used on the Sqoop import.

sqoop import --connect 
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos;database=database_name"

--username 'Domain\user' --query 'select * from table where $CONDITIONS' 
--split-by Abc_Number --target-dir /user/user/Sql

Here is the error:
INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 
ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO 
impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to 
track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job: 
Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce 
0% INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0, 
Status : FAILED Error: java.lang.RuntimeException: 
java.lang.RuntimeException: 
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication 
failed. ClientConnectionId:366c38c7-c at 
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) 
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) 
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) 
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: 
java.lang.RuntimeException: 
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication 
failed. ClientConnectionId:366c38c7-c at 
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) 
at 
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) 
... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: 
Integrated authentication failed. ClientConnectionId:366c38c7-c at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)

at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176) 
at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159) 
at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42) 
at 
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)

at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627) 
at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772) 
at 
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168) 
at java.sql.DriverManager.getConnection(DriverManager.java:664) at 
java.sql.DriverManager.getConnection(DriverManager.java:247) at 
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) 
at 
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) 
... 10 more Caused by: java.security.PrivilegedActionException: 
GSSException: No valid credentials provided (Mechanism level: Failed to 
find any Kerberos tgt) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)

at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150) 
... 26 more Caused by: GSSException: No valid credentials provided 
(Mechanism level: Failed to find any Kerberos tgt) at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) 
at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) 
at 
sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) 
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at 
sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at 
sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) 
at 
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189) 
at 
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187) 
... 30 more


Does anyone know if Scoop support forwarding the Kerberos ticket? and if 
so, what do we need to configure to have it do so? Any comments on how 
this should work, or websites discussing how you got this working in your 
environment?

There is a Hortonworks Community post on this as well:  
https://community.hortonworks.com/questions/52137/sqlserver-sqoop-import-integrated-security-kerbero.html

I would like to verify each step of the process can support Kerberos, and 
then verify the configurations to determine if there is any configuration 
set that might prevent this from working.

Any help would be appreciated.

thanks.

Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
          (414-357-4062) | jkaufman@footlocker.com


Visit us on-line at footlocker.com. 

The information in this e-mail, and any attachment therein, is 
confidential and for use by the addressee only. If you are not the 
intended recipient, please return the e-mail to the sender and delete it 
from your computer. Although the Company attempts to sweep e-mail and 
attachments for viruses, it does not guarantee that either are virus-free 
and accepts no liability for any damage sustained as a result of viruses.
Mime
View raw message